{"rsdb":{"rid":"309135","subhead":"","postdate":"0","aid":"224297","fid":"116","uid":"1","topic":"1","content":"<div id=\"article_content\" class=\"article_content clearfix csdn-tracking-statistics\" data-pid=\"blog\" data-mod=\"popu_307\" data-dsm=\"post\">\n            <link rel=\"stylesheet\" href=\"https:\/\/csdnimg.cn\/release\/phoenix\/template\/css\/ck_htmledit_views-f57960eb32.css\">\n                              <link rel=\"stylesheet\" href=\"https:\/\/csdnimg.cn\/release\/phoenix\/template\/css\/ck_htmledit_views-f57960eb32.css\">\n          <div class=\"htmledit_views\" id=\"content_views\">\n            <div class=\"content-detail markdown-body\">\n                    \n<div>\n<div class=\"title___3ntsK\">\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:28pt;line-height:83%;\">MaxCompute Spark<\/span><span style=\"font-size:28pt;line-height:83%;\">\u5f00\u53d1\u6307\u5357<\/span><\/h1>\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:28pt;line-height:83%;\"><br><\/span><\/h1>\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:28pt;line-height:83%;\">0. <\/span><span style=\"font-size:28pt;line-height:83%;\">\u6982\u8ff0<\/span>\n<\/h1>\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><br><\/h1>\n<p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;\">\u672c\u6587\u6863\u9762\u5411\u9700\u8981\u4f7f\u7528<span>MaxCompute Spark<\/span>\u8fdb\u884c\u5f00\u53d1\u7684\u7528\u6237\u4f7f\u7528\u3002\u672c\u6307\u5357\u4e3b\u8981\u9002\u7528\u4e8e\u5177\u5907\u6709<span>Spark<\/span>\u5f00\u53d1\u7ecf\u9a8c\u7684\u5f00\u53d1\u4eba\u5458\u3002<br><\/p>\n<p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;\"><span>MaxCompute Spark<\/span>\u662f<span>MaxCompute<\/span>\u63d0\u4f9b\u7684\u517c\u5bb9\u5f00\u6e90\u7684<span>Spark<\/span>\u8ba1\u7b97\u670d\u52a1\uff0c\u5b83\u5728\u7edf\u4e00\u7684\u8ba1\u7b97\u8d44\u6e90\u548c\u6570\u636e\u96c6\u6743\u9650\u4f53\u7cfb\u4e4b\u4e0a\uff0c\u63d0\u4f9b<span>Spark<\/span>\u8ba1\u7b97\u6846\u67b6\uff0c\u652f\u6301\u7528\u6237\u4ee5\u719f\u6089\u7684\u5f00\u53d1\u4f7f\u7528\u65b9\u5f0f\u63d0\u4ea4\u8fd0\u884c<span>Spark<\/span>\u4f5c\u4e1a\uff0c\u4ee5\u6ee1\u8db3\u66f4\u4e30\u5bcc\u7684\u6570\u636e\u5904\u7406\u5206\u6790\u573a\u666f\u3002<\/p>\n<p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;\">\u672c\u6587\u5c06\u91cd\u70b9\u4ecb\u7ecd<span>MaxCompute Spark<\/span>\u80fd\u591f\u652f\u6491\u7684\u5e94\u7528\u573a\u666f\uff0c\u540c\u65f6\u8bf4\u660e\u5f00\u53d1\u7684\u4f9d\u8d56\u6761\u4ef6\u548c\u73af\u5883\u51c6\u5907\uff0c\u91cd\u70b9\u5bf9<span>Spark<\/span>\u4f5c\u4e1a\u5f00\u53d1\u3001\u63d0\u4ea4\u5230<span>MaxCompute<\/span>\u96c6\u7fa4\u6267\u884c\u3001\u8bca\u65ad\u8fdb\u884c\u4ecb\u7ecd\u3002<\/p>\n<div><br><\/div>\n\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:28pt;line-height:83%;\">1. <\/span><span style=\"font-size:28pt;line-height:83%;\">\u524d\u63d0\u6761\u4ef6<\/span>\n<\/h1>\n<div><span style=\"font-size:28pt;line-height:83%;\"><br><\/span><\/div>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p style=\"line-height:24px;letter-spacing:.7px;font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;\">MaxCompute Spark\u662f\u963f\u91cc\u4e91\u63d0\u4f9b\u7684Spark on MaxCompute\u7684\u89e3\u51b3\u65b9\u6848\uff0c\u80fd\u591f\u8ba9Spark\u5e94\u7528\u8fd0\u884c\u5728\u6258\u7ba1\u7684MaxCompute\u8ba1\u7b97\u73af\u5883\u4e2d\u3002\u4e3a\u4e86\u80fd\u591f\u5728MaxCompute\u73af\u5883\u4e2d\u5b89\u5168\u5730\u8fd0\u884cSpark\u4f5c\u4e1a\uff0cMaxCompute\u63d0\u4f9b\u4e86\u4ee5\u4e0bSDK\u548cMaxCompute Spark\u5b9a\u5236\u53d1\u5e03\u5305\u3002<\/p>\n<p style=\"line-height:24px;letter-spacing:.7px;font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;\"><span style=\"letter-spacing:.7px;\">SDK\u5b9a\u4f4d\u4e8e<\/span><span style=\"letter-spacing:.7px;\">\u5f00\u6e90\u5e94\u7528\u63a5\u5165MaxCompute SDK\uff1a<\/span><br><span style=\"letter-spacing:.7px;text-indent:2em;\">\u63d0\u4f9b\u4e86\u96c6\u6210\u6240\u9700\u7684API\u8bf4\u660e\u4ee5\u53ca\u76f8\u5173\u529f\u80fdDemo\uff0c\u7528\u6237\u53ef\u4ee5\u57fa\u4e8e\u9879\u76ee\u63d0\u4f9b\u7684Spark-1.x\u4ee5\u53caSpark-2.x\u7684example\u9879\u76ee\u6784\u5efa\u81ea\u5df1\u7684\u5e94\u7528\uff0c\u5e76\u4e14\u63d0\u4ea4\u5230MaxCompute\u96c6\u7fa4\u4e0a<\/span><span style=\"letter-spacing:.7px;text-indent:2em;\">\u3002<\/span><br><span style=\"letter-spacing:.7px;text-indent:2em;\">MaxCompute Spark\u5ba2\u6237\u7aef\u53d1\u5e03\u5305\uff1a<\/span><br><span style=\"letter-spacing:.7px;text-indent:2em;\">\u96c6\u6210\u4e86MaxCompute\u8ba4\u8bc1\u529f\u529f\u80fd\uff0c\u4f5c\u4e3a\u5ba2\u6237\u7aef\u5de5\u5177\uff0c\u7528\u4e8e\u901a\u8fc7Spark-submit\u65b9\u5f0f\u63d0\u4ea4\u4f5c\u4e1a\u5230MaxCompute\u9879\u76ee\u4e2d\u8fd0\u884c\uff0c\u76ee\u524d\u63d0\u4f9b\u4e86\u9762\u5411Spark1.x\u548cSpark2.x\u76842\u4e2a\u53d1\u5e03\u5305:spark-1.6.3\u548cspark-2.3.0 SDK\u5728\u5f00\u53d1\u65f6\uff0c\u53ef\u4ee5\u901a\u8fc7\u914d\u7f6eMaven\u4f9d\u8d56\u8fdb\u884c\u5f15\u7528\u3002Spark\u5ba2\u6237\u7aef\u9700\u8981\u6839\u636e\u5f00\u53d1\u7684Spark\u7248\u672c\uff0c\u63d0\u524d\u4e0b\u8f7d\u3002\u5982\uff0c\u9700\u8981\u5f00\u53d1Spark1.x\u5e94\u7528\uff0c\u5e94\u4e0b\u8f7dspark-1.6.3\u7248\u672c\u5ba2\u6237\u7aef\uff1b\u5982\u9700\u5f00\u53d1Spark2.x\u5e94\u7528\uff0c\u5e94\u4e0b\u8f7dspark-2.3.0\u5ba2\u6237\u7aef\u3002<\/span><br><\/p>\n<p><\/p>\n\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:28pt;line-height:83%;\">2. <\/span><span style=\"font-size:28pt;line-height:83%;\">\u5f00\u53d1\u73af\u5883\u51c6\u5907<\/span>\n<\/h1>\n<div><span style=\"font-size:28pt;line-height:83%;\"><br><\/span><\/div>\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">2.1 Maxcompute\nSpark<\/span><span style=\"font-size:24pt;line-height:83%;\">\u5ba2\u6237\u7aef\u51c6\u5907<\/span>\n<\/h2>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><\/p>\n<p><\/p>\n<p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-style:normal;font-weight:400;text-indent:0px;text-transform:none;word-spacing:0px;\">MaxCompute Spark\u53d1\u5e03\u5305\uff1a\u96c6\u6210\u4e86MaxCompute\u8ba4\u8bc1\u529f\u529f\u80fd\uff0c\u4f5c\u4e3a\u5ba2\u6237\u7aef\u5de5\u5177\uff0c\u7528\u4e8e\u901a\u8fc7Spark-submit\u65b9\u5f0f\u63d0\u4ea4\u4f5c\u4e1a\u5230MaxCompute\u9879\u76ee\u4e2d\u8fd0\u884c\uff0c\u76ee\u524d\u63d0\u4f9b\u4e86\u9762\u5411Spark1.x\u548cSpark2.x\u76842\u4e2a\u53d1\u5e03\u5305\uff1a<\/p>\n<ul><li class=\"MsoNormal\"><a href=\"http:\/\/repo.aliyun.com\/download\/spark-1.6.3-public.tar.gz\" rel=\"nofollow\"><span style=\"color:rgb(24,144,255);\">spark-1.6.3<\/span><\/a><\/li>\n<li class=\"MsoNormal\"><a href=\"http:\/\/repo.aliyun.com\/download\/spark-2.3.0-odps0.30.0.tar.gz\" rel=\"nofollow\"><span style=\"color:rgb(24,144,255);\">spark-2.3.0<\/span><\/a><\/li>\n<\/ul><p style=\"line-height:24px;letter-spacing:.7px;font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;\">\u8bf7\u6839\u636e\u9700\u8981\u5f00\u53d1\u7684Spark\u7248\u672c\uff0c\u9009\u62e9\u5408\u9002\u7684\u7248\u672c\u4e0b\u8f7d\u5e76\u89e3\u538bMaxcompute Spark\u53d1\u5e03\u5305\u3002<\/p>\n<div><br><\/div>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">2.2 <\/span><span style=\"font-size:24pt;line-height:83%;\">\u8bbe\u7f6e\u73af\u5883\u53d8\u91cf<\/span>\n<\/h2>\n<div>\n<span style=\"font-size:24pt;line-height:83%;\">\n<\/span><p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">JAVA_HOME\u8bbe\u7f6e<\/p>\n<p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;\"><br><\/span><\/p>\n<\/div>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\"># \u5c3d\u91cf\u4f7f\u7528JDK 1.7+ 1.8+ \u6700\u4f73<br>\nexport JAVA_HOME=\/path\/to\/jdk<br>\nexport CLASSPATH=.:$JAVA_HOME\/lib\/dt.jar:$JAVA_HOME\/lib\/tools.jar<br>\nexport PATH=$JAVA_HOME\/bin:$PATH<\/p>\n<\/div>\n<p>SPARK_HOME\u8bbe\u7f6e<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">export\nSPARK_HOME=\/path\/to\/spark_extracted_package<br>\nexport PATH=$SPARK_HOME\/bin:$PATH<\/p>\n<\/div>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">2.3 <\/span><span style=\"font-size:24pt;line-height:83%;\">\u8bbe\u7f6e<\/span><span style=\"font-size:24pt;line-height:83%;\">Spark-defaults.conf<\/span>\n<\/h2>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>\u5728 <b>$SPARK_HOME\/conf<\/b>\n<br>\n\u8def\u5f84\u4e0b\u5b58\u5728spark-defaults.conf.template\u6587\u4ef6\uff0c\u8fd9\u4e2a\u53ef\u4ee5\u4f5c\u4e3aspark-defaults.conf\u7684\u6a21\u7248\uff0c\u9700\u8981\u5728\u8be5\u6587\u4ef6\u4e2d\u8bbe\u7f6eMaxCompute\u76f8\u5173\u7684\u8d26\u53f7\u4fe1\u606f\u540e\uff0c\u624d\u53ef\u4ee5\u63d0\u4ea4Spark\u4efb\u52a1\u5230MaxCompute\u3002\u9ed8\u8ba4\u914d\u7f6e\u5185\u5bb9\u5982\u4e0b\uff0c\u5c06\u7a7a\u767d\u90e8\u5206\u6839\u636e\u5b9e\u9645\u7684\u8d26\u53f7\u4fe1\u606f\u586b\u4e0a\u5373\u53ef\uff0c\u5176\u4f59\u7684\u914d\u7f6e\u53ef\u4ee5\u4fdd\u6301\u4e0d\u53d8\u3002<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">#\nMaxCompute\u8d26\u53f7\u4fe1\u606f<br>\nspark.hadoop.odps.project.name =<br>\nspark.hadoop.odps.access.id =<br>\nspark.hadoop.odps.access.key =<br>\n# \u4ee5\u4e0b\u914d\u7f6e\u4fdd\u6301\u4e0d\u53d8<br>\nspark.sql.catalogImplementation=odps<br>\nspark.hadoop.odps.task.major.version = cupid_v2<br>\nspark.hadoop.odps.cupid.container.image.enable = true<br>\nspark.hadoop.odps.cupid.container.vm.engine.type = hyper<br>\nspark.hadoop.odps.end.point = http:\/\/service.cn.maxcompute.aliyun.com\/api<br>\nspark.hadoop.odps.runtime.end.point =\nhttp:\/\/service.cn.maxcompute.aliyun-inc.com\/api<\/p>\n<\/div>\n\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:28pt;line-height:83%;\">3. <\/span><span style=\"font-size:28pt;line-height:83%;\">\u8bbf\u95ee<\/span><span style=\"font-size:28pt;line-height:83%;\">MaxCompute<\/span><span style=\"font-size:28pt;line-height:83%;\">\u8868\u6240\u9700\u4f9d\u8d56<\/span>\n<\/h1>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>\u82e5\u4f5c\u4e1a\u9700\u8981\u8bbf\u95eeMaxCompute\u8868\uff0c\u9700\u8981\u4f9d\u8d56odps-spark-datasource\u6a21\u5757\uff0c\u672c\u8282\u4ecb\u7ecd\u5982\u4f55\u628a\u8be5\u4f9d\u8d56\u7f16\u8bd1\u5b89\u88c5\u5230\u672c\u5730maven\u4ed3\u5e93\uff1b\u82e5\u65e0\u9700\u8bbf\u95ee\u53ef\u76f4\u63a5\u8df3\u8fc7\u3002<\/p>\n<ol><li class=\"MsoNormal\">git clone\u4ee3\u7801\uff0cgithub\u5730\u5740:<a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/tree\/3.3.2-public\" rel=\"nofollow\"><span style=\"color:rgb(24,144,255);\">https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/tree\/3.3.2-public<\/span><\/a>\n<\/li>\n<\/ol><p><\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">#git\nclone git@github.com:aliyun\/aliyun-cupid-sdk.git<\/p>\n<\/div>\n<p><\/p>\n<ol><li class=\"MsoNormal\">\u7f16\u8bd1\u6a21\u5757<\/li>\n<\/ol><div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">#cd\n${path to aliyun-cupid-sdk}<br>\n#git checkout 3.3.2-public<br><br>\n\/\/ \u7f16\u8bd1\u5e76\u5b89\u88c5cupid-sdk<br>\n#cd ${path to aliyun-cupid-sdk}\/core\/cupid-sdk\/<br>\n#mvn clean install -DskipTests<br><br>\n\/\/ \u7f16\u8bd1\u5e76\u5b89\u88c5datasource\u3002\u4f9d\u8d56cupid-sdk<br>\n\/\/ for spark-2.x<br>\n# cd ${path to aliyun-cupid-sdk}\/spark\/spark-2.x\/datasource<br>\n# mvn clean install -DskipTests<br>\n\/\/ for spark-1.x<br>\n# cd ${path to aliyun-cupid-sdk}\/spark\/spark-1.x\/datasource<br>\n#mvn clean install -DskipTests<\/p>\n<\/div>\n<p><\/p>\n<ol><li class=\"MsoNormal\">\u6dfb\u52a0\u4f9d\u8d56<\/li>\n<\/ol><div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">&lt;!--\nSpark-1.x\u8bf7\u4f9d\u8d56\u6b64\u6a21\u5757 --&gt;<br>\n&lt;dependency&gt;<br>\n&lt;groupId&gt;com.aliyun.odps&lt;\/groupId&gt;<br>\n&lt;artifactId&gt;odps-spark-datasource_2.10&lt;\/artifactId&gt;<br>\n&lt;version&gt;3.3.2-public&lt;\/version&gt;<br>\n&lt;\/dependency&gt;<br><br>\n&lt;!-- Spark-2.x\u8bf7\u4f9d\u8d56\u6b64\u6a21\u5757 --&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;com.aliyun.odps&lt;\/groupId&gt;<br>\n\n&lt;artifactId&gt;odps-spark-datasource_2.11&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;3.3.2-public&lt;\/version&gt;<br>\n&lt;\/dependency&gt;<\/p>\n<\/div>\n<p><\/p>\n\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:28pt;line-height:83%;\">4. OSS<\/span><span style=\"font-size:28pt;line-height:83%;\">\u4f9d\u8d56<\/span>\n<\/h1>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>\u82e5\u4f5c\u4e1a\u9700\u8981\u8bbf\u95eeOSS\uff0c\u76f4\u63a5\u6dfb\u52a0\u4ee5\u4e0b\u4f9d\u8d56\u5373\u53ef<\/p>\n<p><\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">&lt;dependency&gt;<br>\n\n&lt;groupId&gt;com.aliyun.odps&lt;\/groupId&gt;<br>\n\n&lt;artifactId&gt;hadoop-fs-oss&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;3.3.2-public&lt;\/version&gt;<br>\n&lt;\/dependency&gt;<\/p>\n<\/div>\n<p><\/p>\n\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:11pt;line-height:83%;\">5. <\/span><span style=\"font-size:11pt;line-height:83%;\">\u5e94\u7528\u5f00\u53d1<\/span>\n<\/h1>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>MaxCompute\u4ea7\u54c1\u63d0\u4f9b\u4e86\u4e24\u4e2a\u5e94\u7528\u6784\u5efa\u7684\u6a21\u7248\uff0c\u7528\u6237\u53ef\u4ee5\u57fa\u4e8e\u6b64\u6a21\u7248\u8fdb\u884c\u5f00\u53d1\uff0c\u6700\u540e\u7edf\u4e00\u6784\u5efa\u6574\u4e2a\u9879\u76ee\u540e\u7528\u751f\u6210\u7684\u5e94\u7528\u5305\u5373\u53ef\u76f4\u63a5\u63d0\u4ea4\u5230MaxCompute\u96c6\u7fa4\u4e0a\u8fd0\u884cSpark\u5e94\u7528\u3002<\/p>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">5.1 <\/span><span style=\"font-size:24pt;line-height:83%;\">\u901a\u8fc7\u6a21\u7248\u6784\u5efa\u5e94\u7528<\/span>\n<\/h2>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>MaxCompute Spark\u63d0\u4f9b\u4e24\u4e2a\u5e94\u7528\u6784\u5efa\u6a21\u7248\uff0c\u7528\u6237\u53ef\u4ee5\u57fa\u4e8e\u6b64\u6a21\u7248\u8fdb\u884c\u5f00\u53d1\uff0c\u6700\u540e\u7edf\u4e00\u6784\u5efa\u6574\u4e2a\u9879\u76ee\u540e\u7528\u751f\u6210\u7684\u5e94\u7528\u5305\u5373\u53ef\u76f4\u63a5\u63d0\u4ea4\u5230MaxCompute\u96c6\u7fa4\u4e0a\u8fd0\u884cSpark\u5e94\u7528\u3002\u9996\u5148\u9700\u8981\u628a\u4ee3\u7801clone\u4e0b\u6765<\/p>\n<p><\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">#git\nclone git@github.com:aliyun\/aliyun-cupid-sdk.git<br>\n#cd aliyun-cupid-sdk<br>\n#checkout 3.3.2-public<br>\n#cd archetypes<br><br>\n\/\/ for Spark-1.x<br>\nsh Create-AliSpark-1.x-APP.sh spark-1.x-demo \/tmp<br><br>\n\/\/ for Spark-2.x<br>\nCreate-AliSpark-2.x-APP.sh spark-2.x-demo \/tmp<\/p>\n<\/div>\n<p>\u4ee5\u4e0a\u547d\u4ee4\u4f1a\u5728\/tmp\u76ee\u5f55\u4e0b\u521b\u5efa\u540d\u4e3a spark-1.x-demo(spark-2.x-demo)\u7684maven project\uff0c\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\u8fdb\u884c\u7f16\u8bd1\u548c\u63d0\u4ea4\u4f5c\u4e1a:<\/p>\n<p><\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">#cd\n\/tmp\/spark-2.x\/demo<br>\n#mvn clean package<br><br>\n\/\/ \u63d0\u4ea4\u4f5c\u4e1a<br>\n$SPARK_HOME\/bin\/spark-submit \\<br>\n--master yarn-cluster \\<br>\n--class SparkPi \\<br>\n\/tmp\/spark-2.x-demo\/target\/AliSpark-2.x-quickstart-1.0-SNAPSHOT-shaded.jar<\/p>\n<\/div>\n<p><\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">#\nUsage: sh Create-AliSpark-2.x-APP.sh &lt;app_name&gt; &lt;target_path&gt;<br>\nsh Create-AliSpark-2.x-APP.sh spark-2.x-demo \/tmp\/<br>\ncd \/tmp\/spark-2.x-demo<br>\nmvn clean package<br>\n# \u5192\u70df\u6d4b\u8bd5 <br>\n# 1 \u5229\u7528\u7f16\u8bd1\u51fa\u6765\u7684shaded jar\u5305<br>\n# 2 \u6309\u7167\u6587\u6863\u6240\u793a\u4e0b\u8f7dMaxCompute Spark\u5ba2\u6237\u7aef<br>\n# 3 \u53c2\u8003\u6587\u6863\u201d\u7f6e\u73af\u5883\u53d8\u91cf\u201d\u6307\u5f15\uff0c\u586b\u5199MaxCompute\u9879\u76ee\u76f8\u5173\u914d\u7f6e\u9879<br>\n# \u6267\u884cspark-submit\u547d\u4ee4 \u5982\u4e0b<br>\n$SPARK_HOME\/bin\/spark-submit \\<br>\n --master yarn-cluster \\<br>\n --class SparkPi \\<br>\n\n\/tmp\/spark-2.x-demo\/target\/AliSpark-2.x-quickstart-1.0-SNAPSHOT-shaded.jar<\/p>\n<\/div>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">5.2 Java\/Scala<\/span><span style=\"font-size:24pt;line-height:83%;\">\u5f00\u53d1\u6837\u4f8b<\/span>\n<\/h2>\n<h3 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:20pt;line-height:83%;\">Spark-1.x<\/span><\/h3>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>pom.xml \u987b\u77e5<br>\n\u8bf7\u6ce8\u610f \u7528\u6237\u6784\u5efaSpark\u5e94\u7528\u7684\u65f6\u5019\uff0c\u7531\u4e8e\u662f\u7528MaxCompute\u63d0\u4f9b\u7684Spark\u5ba2\u6237\u7aef\u53bb\u63d0\u4ea4\u5e94\u7528\uff0c\u6545\u9700\u8981\u6ce8\u610f\u4e00\u4e9b\u4f9d\u8d56scope\u7684\u5b9a\u4e49<\/p>\n<ul><li class=\"MsoNormal\">spark-core\n     spark-sql\u7b49\u6240\u6709spark\u793e\u533a\u53d1\u5e03\u7684\u5305\uff0c\u7528provided scope<\/li>\n<li class=\"MsoNormal\">odps-spark-datasource\n     \u7528\u9ed8\u8ba4\u7684compile scope<\/li>\n<\/ul><div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">&lt;!--\nspark\u76f8\u5173\u4f9d\u8d56, provided --&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;org.apache.spark&lt;\/groupId&gt;<br>\n\n&lt;artifactId&gt;spark-mllib_${scala.binary.version}&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;${spark.version}&lt;\/version&gt;<br>\n &lt;scope&gt;provided&lt;\/scope&gt;<br>\n&lt;\/dependency&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;org.apache.spark&lt;\/groupId&gt;<br>\n\n&lt;artifactId&gt;spark-sql_${scala.binary.version}&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;${spark.version}&lt;\/version&gt;<br>\n &lt;scope&gt;provided&lt;\/scope&gt;<br>\n&lt;\/dependency&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;org.apache.spark&lt;\/groupId&gt;<br>\n &lt;artifactId&gt;spark-core_${scala.binary.version}&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;${spark.version}&lt;\/version&gt;<br>\n &lt;scope&gt;provided&lt;\/scope&gt;<br>\n&lt;\/dependency&gt;<br><br>\n&lt;!-- datasource\u4f9d\u8d56, \u7528\u4e8e\u8bbf\u95eeMaxCompute\u8868 --&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;com.aliyun.odps&lt;\/groupId&gt;<br>\n &lt;artifactId&gt;odps-spark-datasource_${scala.binary.version}&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;3.3.2-public&lt;\/version&gt;<br>\n&lt;\/dependency&gt;<\/p>\n<\/div>\n<p><\/p>\n\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">\u6848\u4f8b\u8bf4\u660e<\/span><\/h4>\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:11pt;line-height:83%;\">WordCount<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-1.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/WordCount.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">Step\n1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.WordCount \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-1.x\/spark-examples\/target\/spark-examples_2.10-version-shaded.jar<\/p>\n<\/div>\n<p><\/p>\n\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">Spark-SQL on\nMaxCompute Table<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-1.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/sparksql\/SparkSQL.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\"># \u8fd0\u884c\u53ef\u80fd\u4f1a\u62a5Table Not Found\u7684\u5f02\u5e38\uff0c\u56e0\u4e3a\u7528\u6237\u7684MaxCompute\nProject\u4e2d\u6ca1\u6709\u4ee3\u7801\u4e2d\u6307\u5b9a\u7684\u8868<br>\n# \u53ef\u4ee5\u53c2\u8003\u4ee3\u7801\u4e2d\u7684\u5404\u79cd\u63a5\u53e3\uff0c\u5b9e\u73b0\u5bf9\u5e94Table\u7684SparkSQL\u5e94\u7528<br>\nStep 1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.sparksql.SparkSQL \\<br>\n ${path to aliyun-cupid-sdk}\/spark\/spark-1.x\/spark-examples\/target\/spark-examples_2.10-version-shaded.jar<\/p>\n<\/div>\n<p><\/p>\n\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">GraphX PageRank<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-1.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/graphx\/PageRank.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">Step\n1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.graphx.PageRank \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-1.x\/spark-examples\/target\/spark-examples_2.10-version-shaded.jar<\/p>\n<\/div>\n\n<h3 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><\/h3>\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">Mllib\nKmeans-ON-OSS<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-1.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/mllib\/KmeansModelSaveToOss.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\"># \u4ee3\u7801\u4e2d\u7684OSS\u8d26\u53f7\u4fe1\u606f\u76f8\u5173\u9700\u8981\u586b\u4e0a\uff0c\u518d\u7f16\u8bd1\u63d0\u4ea4<br>\nconf.set(\"spark.hadoop.fs.oss.accessKeyId\", \"***\")<br>\nconf.set(\"spark.hadoop.fs.oss.accessKeySecret\", \"***\")<br>\nconf.set(\"spark.hadoop.fs.oss.endpoint\",\n\"oss-cn-hangzhou-zmf.aliyuncs.com\")<br>\nStep 1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.mllib.KmeansModelSaveToOss \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-1.x\/spark-examples\/target\/spark-examples_2.10-version-shaded.jar<\/p>\n<\/div>\n\n<h3 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><\/h3>\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">OSS\nUnstructuredData<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-1.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/oss\/SparkUnstructuredDataCompute.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\"># \u4ee3\u7801\u4e2d\u7684OSS\u8d26\u53f7\u4fe1\u606f\u76f8\u5173\u9700\u8981\u586b\u4e0a\uff0c\u518d\u7f16\u8bd1\u63d0\u4ea4<br>\nconf.set(\"spark.hadoop.fs.oss.accessKeyId\", \"***\")<br>\nconf.set(\"spark.hadoop.fs.oss.accessKeySecret\", \"***\")<br>\nconf.set(\"spark.hadoop.fs.oss.endpoint\",\n\"oss-cn-hangzhou-zmf.aliyuncs.com\")<br>\nStep 1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.oss.SparkUnstructuredDataCompute \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-1.x\/spark-examples\/target\/spark-examples_2.10-version-shaded.jar<\/p>\n<\/div>\n\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><\/h4>\n<h3 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:20pt;line-height:83%;\">Spark-2.x<\/span><\/h3>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>pom.xml \u987b\u77e5<br>\n\u8bf7\u6ce8\u610f \u7528\u6237\u6784\u5efaSpark\u5e94\u7528\u7684\u65f6\u5019\uff0c\u7531\u4e8e\u662f\u7528MaxCompute\u63d0\u4f9b\u7684Spark\u5ba2\u6237\u7aef\u53bb\u63d0\u4ea4\u5e94\u7528\uff0c\u6545\u9700\u8981\u6ce8\u610f\u4e00\u4e9b\u4f9d\u8d56scope\u7684\u5b9a\u4e49<\/p>\n<ul><li class=\"MsoNormal\">spark-core\n     spark-sql\u7b49\u6240\u6709spark\u793e\u533a\u53d1\u5e03\u7684\u5305\uff0c\u7528provided scope<\/li>\n<li class=\"MsoNormal\">odps-spark-datasource\n     \u7528\u9ed8\u8ba4\u7684compile scope<\/li>\n<\/ul><div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">&lt;!--\nspark\u76f8\u5173\u4f9d\u8d56, provided --&gt;<br>\n&lt;dependency&gt;<br>\n &lt;groupId&gt;org.apache.spark&lt;\/groupId&gt;<br>\n\n&lt;artifactId&gt;spark-mllib_${scala.binary.version}&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;${spark.version}&lt;\/version&gt;<br>\n &lt;scope&gt;provided&lt;\/scope&gt;<br>\n&lt;\/dependency&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;org.apache.spark&lt;\/groupId&gt;<br>\n &lt;artifactId&gt;spark-sql_${scala.binary.version}&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;${spark.version}&lt;\/version&gt;<br>\n &lt;scope&gt;provided&lt;\/scope&gt;<br>\n&lt;\/dependency&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;org.apache.spark&lt;\/groupId&gt;<br>\n\n&lt;artifactId&gt;spark-core_${scala.binary.version}&lt;\/artifactId&gt;<br>\n &lt;version&gt;${spark.version}&lt;\/version&gt;<br>\n &lt;scope&gt;provided&lt;\/scope&gt;<br>\n&lt;\/dependency&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;com.aliyun.odps&lt;\/groupId&gt;<br>\n\n&lt;artifactId&gt;cupid-sdk&lt;\/artifactId&gt;<br>\n &lt;scope&gt;provided&lt;\/scope&gt;<br>\n&lt;\/dependency&gt;<br><br>\n&lt;!-- datasource\u4f9d\u8d56, \u7528\u4e8e\u8bbf\u95eeMaxCompute\u8868 --&gt;<br>\n&lt;dependency&gt;<br>\n\n&lt;groupId&gt;com.aliyun.odps&lt;\/groupId&gt;<br>\n\n&lt;artifactId&gt;odps-spark-datasource_${scala.binary.version}&lt;\/artifactId&gt;<br>\n\n&lt;version&gt;3.3.2-public&lt;\/version&gt;<br>\n&lt;\/dependency&gt;<\/p>\n<\/div>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><\/h2>\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">\u6848\u4f8b\u8bf4\u660e<\/span><\/h4>\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">WordCount<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-2.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/WordCount.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">Step\n1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.WordCount \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-2.x\/spark-examples\/target\/spark-examples_2.11-version-shaded.jar<\/p>\n<\/div>\n<p><\/p>\n\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:16pt;line-height:83%;\">Spark-SQL <\/span><span style=\"font-size:16pt;line-height:83%;\">\u64cd\u4f5c<\/span><span style=\"font-size:16pt;line-height:83%;\">MaxCompute<\/span><span style=\"font-size:16pt;line-height:83%;\">\u8868<\/span>\n<\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-2.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/sparksql\/SparkSQL.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\"># \u8fd0\u884c\u53ef\u80fd\u4f1a\u62a5Table Not Found\u7684\u5f02\u5e38\uff0c\u56e0\u4e3a\u7528\u6237\u7684MaxCompute\nProject\u4e2d\u6ca1\u6709\u4ee3\u7801\u4e2d\u6307\u5b9a\u7684\u8868<br>\n# \u53ef\u4ee5\u53c2\u8003\u4ee3\u7801\u4e2d\u7684\u5404\u79cd\u63a5\u53e3\uff0c\u5b9e\u73b0\u5bf9\u5e94Table\u7684SparkSQL\u5e94\u7528<br>\nStep 1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.sparksql.SparkSQL \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-2.x\/spark-examples\/target\/spark-examples_2.11-version-shaded.jar<\/p>\n<\/div>\n<p><\/p>\n\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">GraphX PageRank<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-2.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/graphx\/PageRank.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">Step\n1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.graphx.PageRank \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-2.x\/spark-examples\/target\/spark-examples_2.11-version-shaded.jar<\/p>\n<\/div>\n\n<h3 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><\/h3>\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">Mllib\nKmeans-ON-OSS<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>KmeansModelSaveToOss<br><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-2.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/mllib\/KmeansModelSaveToOss.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\"># \u4ee3\u7801\u4e2d\u7684OSS\u8d26\u53f7\u4fe1\u606f\u76f8\u5173\u9700\u8981\u586b\u4e0a\uff0c\u518d\u7f16\u8bd1\u63d0\u4ea4<br>\nval spark = SparkSession<br>\n .builder()<br>\n\n.config(\"spark.hadoop.fs.oss.accessKeyId\", \"***\")<br>\n .config(\"spark.hadoop.fs.oss.accessKeySecret\",\n\"***\")<br>\n\n.config(\"spark.hadoop.fs.oss.endpoint\",\n\"oss-cn-hangzhou-zmf.aliyuncs.com\")<br>\n\n.appName(\"KmeansModelSaveToOss\")<br>\n .getOrCreate()<br>\nStep 1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.mllib.KmeansModelSaveToOss \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-2.x\/spark-examples\/target\/spark-examples_2.11-version-shaded.jar<\/p>\n<\/div>\n\n<h3 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><\/h3>\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><span style=\"font-size:16pt;line-height:83%;\">OSS\nUnstructuredData<\/span><\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>SparkUnstructuredDataCompute<br><a href=\"https:\/\/github.com\/aliyun\/aliyun-cupid-sdk\/blob\/3.3.2-public\/spark\/spark-2.x\/spark-examples\/src\/main\/scala\/com\/aliyun\/odps\/spark\/examples\/oss\/SparkUnstructuredDataCompute.scala\" rel=\"nofollow\"><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u8be6\u7ec6\u4ee3\u7801<\/span><\/a><br>\n\u63d0\u4ea4\u65b9\u5f0f<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\"># \u4ee3\u7801\u4e2d\u7684OSS\u8d26\u53f7\u4fe1\u606f\u76f8\u5173\u9700\u8981\u586b\u4e0a\uff0c\u518d\u7f16\u8bd1\u63d0\u4ea4<br>\nval spark = SparkSession<br>\n .builder()<br>\n\n.config(\"spark.hadoop.fs.oss.accessKeyId\", \"***\")<br>\n\n.config(\"spark.hadoop.fs.oss.accessKeySecret\",\n\"***\")<br>\n\n.config(\"spark.hadoop.fs.oss.endpoint\",\n\"oss-cn-hangzhou-zmf.aliyuncs.com\")<br>\n .appName(\"SparkUnstructuredDataCompute\")<br>\n .getOrCreate()<br>\nStep 1. build aliyun-cupid-sdk<br>\nStep 2. properly set spark.defaults.conf<br>\nStep 3. bin\/spark-submit --master yarn-cluster --class \\<br>\n\ncom.aliyun.odps.spark.examples.oss.SparkUnstructuredDataCompute \\<br>\n ${path to\naliyun-cupid-sdk}\/spark\/spark-2.x\/spark-examples\/target\/spark-examples_2.11-version-shaded.jar<\/p>\n<\/div>\n\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><\/h4>\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">PySpark<\/span><span style=\"font-size:24pt;line-height:83%;\">\u5f00\u53d1\u6837\u4f8b<\/span>\n<\/h2>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><b>\u9700\u8981\u6587\u4ef6<\/b><br>\n\u82e5\u9700\u8981\u8bbf\u95eeMaxCompute\u8868\uff0c\u5219\u9700\u8981\u53c2\u8003\u7b2c\u4e09\u8282(<b>\u8bbf\u95eeMaxCompute<\/b><b>\u8868\u6240\u9700\u4f9d\u8d56<\/b>)\u7f16\u8bd1datasource\u5305<\/p>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\"><\/h2>\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:16pt;line-height:83%;\">SparkSQL<\/span><span style=\"font-size:16pt;line-height:83%;\">\u5e94\u7528\u793a\u4f8b<\/span><span style=\"font-size:16pt;line-height:83%;\">(spark1.6)<\/span>\n<\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">from\npyspark import SparkContext, SparkConf<br>\nfrom pyspark.sql import OdpsContext<br><br>\nif __name__ == '__main__':<br>\n conf =\nSparkConf().setAppName(\"odps_pyspark\")<br>\n sc = SparkContext(conf=conf)<br>\n sql_context = OdpsContext(sc)<br>\n df = sql_context.sql(\"select id,\nvalue from cupid_wordcount\")<br>\n df.printSchema()<br>\n df.show(200)<br>\n <br>\n df_2 = sql_context.sql(\"select\nid, value from cupid_partition_table1 where pt1 = 'part1'\")<br>\n df_2.show(200)<br><br>\n #Create Drop Table<br>\n sql_context.sql(\"create table\nTestCtas as select * from cupid_wordcount\").show()<br>\n sql_context.sql(\"drop table\nTestCtas\").show()<\/p>\n<\/div>\n<p>\u63d0\u4ea4\u8fd0\u884c:<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">.\/bin\/spark-submit\n\\<br>\n--jars ${path to odps-spark-datasource_2.10-3.3.2-public.jar} \\<br>\nexample.py<\/p>\n<\/div>\n<p><\/p>\n\n<h4 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:16pt;line-height:83%;\">SparkSQL<\/span><span style=\"font-size:16pt;line-height:83%;\">\u5e94\u7528\u793a\u4f8b\uff08<\/span><span style=\"font-size:16pt;line-height:83%;\">spark2.3<\/span><span style=\"font-size:16pt;line-height:83%;\">\uff09<\/span>\n<\/h4>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">from\npyspark.sql import SparkSession<br><br>\nif __name__ == '__main__':<br>\n spark = SparkSession.builder.appName(\"spark\nsql\").getOrCreate()<br><br>\n df = spark.sql(\"select id, value\nfrom cupid_wordcount\")<br>\n df.printSchema()<br>\n df.show(10, 200)<br><br>\n df_2 = spark.sql(\"SELECT\nproduct,category,revenue FROM (SELECT product,category,revenue, dense_rank()\nOVER (PARTITION BY category ORDER BY revenue DESC) as rank FROM productRevenue) tmp WHERE rank &lt;= 2\");<br>\n df_2.printSchema()<br>\n df_2.show(10, 200)<br><br>\n df_3 = spark.sql(\"select id,\nvalue from cupid_partition_table1 where pt1 = 'part1'\")<br>\n df_3.show(10, 200)<br><br>\n #Create Drop Table<br>\n spark.sql(\"create table TestCtas\nas select * from cupid_wordcount\").show()<br>\n spark.sql(\"drop table\nTestCtas\").show()<\/p>\n<\/div>\n<p>\u63d0\u4ea4\u8fd0\u884c:<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">spark-submit\n--master yarn-cluster \\<br>\n--jars ${path to odps-spark-datasource_2.11-3.3.2-public.jar \\<br>\nexample.py<\/p>\n<\/div>\n<p><\/p>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">6. <\/span><span style=\"font-size:24pt;line-height:83%;\">\u901a\u8fc7<\/span><span style=\"font-size:24pt;line-height:83%;\">Spark<\/span><span style=\"font-size:24pt;line-height:83%;\">\u8bbf\u95ee<\/span><span style=\"font-size:24pt;line-height:83%;\">VPC<\/span><span style=\"font-size:24pt;line-height:83%;\">\u73af\u5883\u5185\u670d\u52a1<\/span>\n<\/h2>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>\u5bf9\u4e8e\u7528\u6237\u4f7f\u7528Spark on\nMaxCompute\u5bf9VPC\u73af\u5883\u5185\u7684RDS\u3001Redis\u3001ECS\u4e3b\u673a\u90e8\u7f72\u7684\u670d\u52a1\u7b49\uff0c\u53d7\u9650\u4e8eVPC\u7684\u8bbf\u95ee\u9650\u5236\uff0c\u6682\u65f6\u8fd8\u65e0\u6cd5\u8bbf\u95ee\uff0c\u5373\u5c06\u5728\u8fd1\u671f\u652f\u6301\u3002<\/p>\n<p><\/p>\n\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:28pt;line-height:83%;\">7. <\/span><span style=\"font-size:28pt;line-height:83%;\">\u5982\u4f55\u628a\u5f00\u6e90<\/span><span style=\"font-size:28pt;line-height:83%;\">Spark<\/span><span style=\"font-size:28pt;line-height:83%;\">\u4ee3\u7801\u8fc1\u79fb\u5230<\/span><span style=\"font-size:28pt;line-height:83%;\">Spark on MaxCompute<\/span>\n<\/h1>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p><b>case1. <\/b><b>\u4f5c\u4e1a\u65e0\u9700\u8bbf\u95eeMaxCompute<\/b><b>\u8868\u548cOSS<\/b><br>\n\u7528\u6237jar\u5305\u53ef\u76f4\u63a5\u8fd0\u884c\uff0c\u53c2\u7167\u7b2c\u4e8c\u8282\u51c6\u5907\u5f00\u53d1\u73af\u5883\u548c\u4fee\u6539\u914d\u7f6e\u3002\u6ce8\u610f\uff0c\u5bf9\u4e8espark\u6216hadoop\u7684\u4f9d\u8d56\u5fc5\u987b\u8bbe\u6210provided\u3002<br><b>case2. <\/b><b>\u4f5c\u4e1a\u9700\u8981\u8bbf\u95eeMaxCompute<\/b><b>\u8868<\/b><br>\n\u53c2\u8003\u7b2c\u4e09\u8282\u7f16\u8bd1datasource\u5e76\u5b89\u88c5\u5230\u672c\u5730maven\u4ed3\u5e93\uff0c\u5728pom\u4e2d\u6dfb\u52a0\u4f9d\u8d56\u540e\u91cd\u65b0\u6253\u5305\u5373\u53ef\u3002<br><b>case3. <\/b><b>\u4f5c\u4e1a\u9700\u8981\u8bbf\u95eeOSS<\/b><br>\n\u53c2\u8003\u7b2c\u56db\u8282\u5728pom\u4e2d\u6dfb\u52a0\u4f9d\u8d56\u540e\u91cd\u65b0\u6253\u5305\u5373\u53ef\u3002<\/p>\n<p><\/p>\n\n<h1 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:28pt;line-height:83%;\">8. <\/span><span style=\"font-size:28pt;line-height:83%;\">\u4efb\u52a1\u63d0\u4ea4\u6267\u884c<\/span>\n<\/h1>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>\u76ee\u524dMaxCompute Spark\u652f\u6301\u4ee5\u4e0b\u51e0\u79cd\u8fd0\u884c\u65b9\u5f0f\uff1alocal\u6a21\u5f0f\uff0ccluster\u6a21\u5f0f\uff0c\u548c\u5728DataWorks\u4e2d\u6267\u884c\u6a21\u5f0f\u3002<\/p>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">8.1 Local<\/span><span style=\"font-size:24pt;line-height:83%;\">\u6a21\u5f0f<\/span>\n<\/h2>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>local\u6a21\u5f0f\u4e3b\u8981\u662f\u8ba9\u7528\u6237\u80fd\u591f\u65b9\u4fbf\u7684\u8c03\u8bd5\u5e94\u7528\u4ee3\u7801\uff0c\u4f7f\u7528\u65b9\u5f0f\u8ddf\u793e\u533a\u76f8\u540c\uff0c\u6211\u4eec\u6dfb\u52a0\u4e86\u7528tunnel\u8bfb\u5199ODPS\u8868\u7684\u529f\u80fd\u3002\u7528\u6237\u53ef\u4ee5\u5728ide\u548c\u547d\u4ee4\u884c\u4e2d\u4f7f\u7528\u8be5\u6a21\u5f0f\uff0c\u9700\u8981\u6dfb\u52a0\u914d\u7f6espark.master=local[N]\uff0c\u5176\u4e2dN\u8868\u793a\u6267\u884c\u8be5\u6a21\u5f0f\u6240\u9700\u8981\u7684cpu\u8d44\u6e90\u3002\u6b64\u5916\uff0clocal\u6a21\u5f0f\u4e0b\u7684\u8bfb\u5199\u8868\u662f\u901a\u8fc7\u8bfb\u5199tunnel\u5b8c\u6210\u7684\uff0c\u9700\u8981\u5728Spark-defaults.conf\u4e2d\u589e\u52a0tunnel\u914d\u7f6e\u9879(\u8bf7\u6839\u636eMaxCompute\u9879\u76ee\u6240\u5728\u7684region\u53ca\u7f51\u7edc\u73af\u5883\u586b\u5199\u5bf9\u5e94\u7684<a href=\"https:\/\/help.aliyun.com\/document_detail\/34951.html\" rel=\"nofollow\"><span style=\"color:rgb(24,144,255);\">Tunnel Endpoint<\/span><span style=\"font-family:'\u5b8b\u4f53';color:rgb(24,144,255);\">\u5730\u5740<\/span><\/a>)\uff1atunnel_end_point=http:\/\/dt.cn-beijing.maxcompute.aliyun.com\u3002\u547d\u4ee4\u884c\u6267\u884c\u8be5\u6a21\u5f0f\u7684\u65b9\u5f0f\u5982\u4e0b\uff1a<\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">1.bin\/spark-submit\n--master local[4] \\<br>\n--class com.aliyun.odps.spark.examples.SparkPi \\<br>\n${path to\naliyun-cupid-sdk}\/spark\/spark-2.x\/spark-examples\/target\/spark-examples_2.11-version-shaded.jar<\/p>\n<\/div>\n<p><\/p>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">8.2 Cluster<\/span><span style=\"font-size:24pt;line-height:83%;\">\u6a21\u5f0f<\/span>\n<\/h2>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<p>\u5728Cluster\u6a21\u5f0f\u4e2d\uff0c\u7528\u6237\u9700\u8981\u6307\u5b9a\u81ea\u5b9a\u4e49\u7a0b\u5e8f\u5165\u53e3Main\uff0cMain\u7ed3\u675f\uff08Success or Fail\uff09spark job\u5c31\u4f1a\u7ed3\u675f\u3002\u4f7f\u7528\u573a\u666f\u9002\u5408\u4e8e\u79bb\u7ebf\u4f5c\u4e1a\uff0c\u53ef\u4ee5\u4e0e\u963f\u91cc\u4e91DataWorks\u4ea7\u54c1\u7ed3\u5408\u8fdb\u884c\u4f5c\u4e1a\u8c03\u5ea6\u3002\u547d\u4ee4\u884c\u63d0\u4ea4\u65b9\u5f0f\u5982\u4e0b\uff1a<br><\/p>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"border:none;\">1.bin\/spark-submit\n--master yarn-cluster \\<br>\n\u2013class SparkPi \\<br>\n${ProjectRoot}\/spark\/spark-2.x\/spark-examples\/target\/spark-examples_2.11-version-shaded.jar<\/p>\n<\/div>\n<p><\/p>\n\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<span style=\"font-size:24pt;line-height:83%;\">8.3 DataWorks<\/span><span style=\"font-size:24pt;line-height:83%;\">\u6267\u884c\u6a21\u5f0f<\/span>\n<\/h2>\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<\/h2>\n<p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;\">\u7528\u6237\u53ef\u4ee5\u5728DataWorks\u4e2d\u8fd0\u884cMaxCompute Spark\u79bb\u7ebf\u4f5c\u4e1a\uff08cluster\u6a21\u5f0f\uff09\uff0c\u4ee5\u65b9\u4fbf\u4e0e\u5176\u4ed6\u7c7b\u578b\u6267\u884c\u8282\u70b9\u96c6\u6210\u548c\u8c03\u5ea6\u3002<\/p>\n<h2 style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\u7528\u6237\u9700\u8981\u5728DataWorks\u7684\u4e1a\u52a1\u6d41\u7a0b\u4e2d\u4e0a\u4f20\u5e76\u63d0\u4ea4(\u8bb0\u5f97\u8981\u5355\u51fb\"\u63d0\u4ea4\"\u6309\u94ae)\u8d44\u6e90\uff1a<\/h2>\n<h1 style=\"line-height:1.389;\">\n<\/h1>\n<div style=\"text-align:left;\"><img src=\"https:\/\/yqfile.alicdn.com\/99b60403daa44d5eb6ae53a8db05e8fe0a3ec828.png\" width=\"300\" alt=\"99b60403daa44d5eb6ae53a8db05e8fe0a3ec828\"><\/div>\n<div style=\"text-align:left;\">\n<img src=\"https:\/\/yqfile.alicdn.com\/2b4ffa687081723046b1f0a2e4d42ccc0b51d62e.png\" width=\"300\" alt=\"2b4ffa687081723046b1f0a2e4d42ccc0b51d62e\"><br><\/div>\n\n<div>\n<span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;letter-spacing:.7px;\">\u7b2c\u4e8c\u6b65\uff1a\u5728\u521b\u5efa\u7684\u4e1a\u52a1\u6d41\u7a0b\u4e2d\uff0c\u4ece<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;letter-spacing:.7px;\">\u6570\u636e\u5f00\u53d1<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;letter-spacing:.7px;\">\u7ec4\u4ef6\u4e2d\u9009\u62e9<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;letter-spacing:.7px;\">ODPS Spark<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;letter-spacing:.7px;\">\u8282\u70b9\u3002<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;letter-spacing:.7px;\"><br><\/span>\n<\/div>\n<h1 style=\"line-height:1.389;\">\n<img src=\"https:\/\/yqfile.alicdn.com\/ea16e393ef93173049ee33c2855f39b0c08ed2cf.png\" width=\"300\" alt=\"ea16e393ef93173049ee33c2855f39b0c08ed2cf\"><br>\n\u53cc\u51fb\u62d6\u62fd\u5230\u5de5\u4f5c\u6d41\u7684Spark\u8282\u70b9\uff0c\u5bf9Spark\u4f5c\u4e1a\u8fdb\u884c\u4efb\u52a1\u5b9a\u4e49\uff1a<br><span style=\"font-size:11pt;color:rgb(38,38,38);\">\n    <img src=\"https:\/\/yqfile.alicdn.com\/6a8184055541d90c294a2137031a88bbb7f6453d.png\" width=\"300\" alt=\"6a8184055541d90c294a2137031a88bbb7f6453d\"><\/span><br><\/h1>\n<h1 style=\"line-height:1.389;\">\n<font>\u9009\u62e9Spark\u7684\u7248\u672c\u3001\u4efb\u52a1\u4f7f\u7528\u7684\u5f00\u53d1\u8bed\u8a00\uff0c\u5e76\u6307\u5b9a\u4efb\u52a1\u6240\u4f7f\u7528\u7684\u8d44\u6e90\u6587\u4ef6\u3002\u8fd9\u91cc\u7684\u8d44\u6e90\u6587\u4ef6\u5c31\u662f\u7b2c\u4e00\u6b65\u5728\u4e1a\u52a1\u6d41\u7a0b\u4e2d\u9884\u5148\u4e0a\u4f20\u5e76\u53d1\u5e03\u7684\u8d44\u6e90\u6587\u4ef6\u3002\u540c\u65f6\uff0c\u60a8\u8fd8\u53ef\u4ee5\u6307\u5b9a\u63d0\u4ea4\u4f5c\u4e1a\u65f6\u7684\u914d\u7f6e\u9879\uff0c\u5982executor\u7684\u6570\u91cf\u3001\u5185\u5b58\u5927\u5c0f\u7b49\u914d\u7f6e\u9879\u3002\u540c\u65f6\u8bbe\u7f6e\u914d\u7f6e\u9879\uff1a<code style=\"border:1px solid;line-height:inherit;color:rgb(38,38,38);font-weight:400;letter-spacing:.7px;\">spark.hadoop.odps.cupid.webproxy.endpoint<\/code><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;letter-spacing:.7px;\">(\u53d6\u503c\u586b\u5199\u9879\u76ee\u6240\u5728region\u7684endpoint\uff0c\u5982<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;letter-spacing:.7px;\">http:\/\/service.cn.maxcompute.aliyun-inc.com\/api<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;letter-spacing:.7px;\">)\u3001<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;letter-spacing:.7px;\">spark.hadoop.odps.moye.trackurl.host<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;letter-spacing:.7px;\">(\u53d6\u503c\u586b\u5199\uff1a<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;letter-spacing:.7px;\">http:\/\/jobview.odps.aliyun.com<\/span><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;letter-spacing:.7px;\">)<\/span><br style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-weight:400;letter-spacing:.7px;\"><\/font><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;letter-spacing:.7px;\">\u4ee5\u4fbf\u80fd\u591f\u67e5\u770b\u65e5\u5fd7\u4e2d\u6253\u5370\u51fa\u7684jobview\u4fe1\u606f\u3002<\/span><br style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;letter-spacing:.7px;\"><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;letter-spacing:.7px;\">\u624b\u52a8\u6267\u884cSpark\u8282\u70b9\uff0c\u53ef\u4ee5\u67e5\u770b\u8be5\u4efb\u52a1\u7684\u6267\u884c\u65e5\u5fd7\uff0c\u4ece\u6253\u5370\u51fa\u6765\u7684\u65e5\u5fd7\u4e2d\u53ef\u4ee5\u83b7\u53d6\u8be5\u4efb\u52a1\u7684logview\u548cjobview\u7684url\uff0c\u7f16\u8bd1\u8fdb\u4e00\u6b65\u67e5\u770b\u4e0e\u8bca\u65ad<\/span>\n<\/h1>\n<div><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;letter-spacing:.7px;\"><img src=\"https:\/\/yqfile.alicdn.com\/23962e5587074a9e83e1845f78574788cc3d326a.png\" width=\"300\" alt=\"23962e5587074a9e83e1845f78574788cc3d326a\"><br><\/span><\/div>\n<div><span style=\"color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;letter-spacing:.7px;\"><img src=\"https:\/\/yqfile.alicdn.com\/6da90f727f550a076bceb2213fd1a9a266aa960e.png\" width=\"300\" alt=\"6da90f727f550a076bceb2213fd1a9a266aa960e\"><br><\/span><\/div>\n<h1 style=\"line-height:1.389;\">Spark\u4f5c\u4e1a\u5b9a\u4e49\u5b8c\u6210\u540e\uff0c\u5373\u53ef\u4ee5\u5728\u4e1a\u52a1\u6d41\u7a0b\u4e2d\u5bf9\u4e0d\u540c\u7c7b\u578b\u670d\u52a1\u8fdb\u884c\u7f16\u6392\u3001\u7edf\u4e00\u8c03\u5ea6\u6267\u884c\u3002\n<\/h1>\n<p style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;\"><\/p>\n\n<h1 style=\"color:rgb(38,38,38);word-spacing:1px;font-size:28px;line-height:36px;font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;letter-spacing:.7px;\">9. \u4f5c\u4e1a\u8bca\u65ad<\/h1>\n<div>\n<p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\u63d0\u4ea4\u4f5c\u4e1a\u540e\uff0c\u9700\u8981\u6839\u636e\u4f5c\u4e1a\u65e5\u5fd7\u6765\u68c0\u67e5\u4f5c\u4e1a\u662f\u5426\u6b63\u5e38\u63d0\u4ea4\u5e76\u6267\u884c\uff0cMaxCompute\u5bf9\u4e8eSpark\u4f5c\u4e1a\u63d0\u4f9b\u4e86Logview\u5de5\u5177\u4ee5\u53caSpark Web-UI\u6765\u5e2e\u52a9\u5f00\u53d1\u8005\u8fdb\u884c\u4f5c\u4e1a\u8bca\u65ad\u3002<\/p>\n<p style=\"line-height:24px;letter-spacing:.7px;color:rgb(38,38,38);font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\u4f8b\u5982\uff0c\u901a\u8fc7Spark-submit\u65b9\u5f0f(dataworks\u6267\u884cspark\u4efb\u52a1\u65f6\u4e5f\u4f1a\u4ea7\u751f\u76f8\u5e94\u65e5\u5fd7)\u63d0\u4ea4\u4f5c\u4e1a\uff0c\u5728\u4f5c\u4e1a\u65e5\u5fd7\u4e2d\u4f1a\u6253\u5370\u4ee5\u4e0b\u5173\u952e\u5185\u5bb9\uff1a<\/p>\n<\/div>\n<h1 style=\"color:rgb(38,38,38);font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;line-height:1.389;\">\n<\/h1>\n<div style=\"border:solid #E8E8E8 1pt;background:#FAFAFA;\">\n<p style=\"line-height:24px;letter-spacing:.7px;font-family:'Tahoma For Number', 'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;font-weight:400;\"><span style=\"font-family:'Chinese Quote', '-apple-system', BlinkMacSystemFont, 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:36px;\">cd\n$SPARK_HOME<\/span><br><\/p>\n<p style=\"border:none;\">\nbin\/spark-submit --master yarn-cluster --class\nSparkPi\n\/tmp\/spark-2.x-demo\/target\/AliSpark-2.x-quickstart-1.0-SNAPSHOT-shaded.jar<br>\n\u4f5c\u4e1a\u63d0\u4ea4\u6210\u529f\u540e\uff0cMaxCompute\u4f1a\u521b\u5efa\u4e00\u4e2ainstance\uff0c\u5728\u65e5\u5fd7\u4e2d\u4f1a\u6253\u5370instance\u7684logview:<br>\n19\/01\/05 20:36:47 INFO YarnClientImplUtil: logview url: http:\/\/logview.odps.aliyun.com\/logview\/h=http:\/\/service.cn.maxcompute.aliyun.com\/api&amp;p=qn_beijing&amp;i=20190105123647703gpqn26pr2&amp;token=eG94TG1iTkZDSFErc1ZPcUZyTTdSWWQ3UE44PSxPRFBTX09CTzoxODc1NjUzNjIyNTQzMDYxLDE1NDY5NTEwMDcseyJTdGF0ZW1lbnQiOlt7IkFjdGlvbiI6WyJvZHBzOlJlYWQiXSwiRWZmZWN0IjoiQWxsb3ciLCJSZXNvdXJjZSI6WyJhY3M6b2RwczoqOnByb2plY3RzL3FuX2JlaWppbmcvaW5zdGFuY2VzLzIwMTkwMTA1MTIzNjQ3NzAzZ3BxbjI2cHIyIl19XSwiVmVyc2lvbiI6IjEifQ==<br>\n\u6210\u529f\u6807\u51c6: &lt;\u770b\u5230\u4ee5\u4e0b\u8f93\u51fa\uff0c\u53ef\u80fd\u4f1a\u6709\u5176\u4ed6\u65e5\u5fd7\u4e00\u5e76\u8f93\u51fa&gt;<br>\n19\/01\/05 20:37:34 INFO Client: <br>\nclient token: N\/A<br>\ndiagnostics: N\/A<br>\nApplicationMaster host: 11.220.203.36<br>\nApplicationMaster RPC port: 30002<br>\nqueue: queue<br>\nstart time: 1546691807945<br>\nfinal status: SUCCEEDED<br>\ntracking URL:\nhttp:\/\/jobview.odps.aliyun.com\/proxyview\/jobview\/h=http:\/\/service.cn.maxcompute.aliyun-inc.com\/api&amp;p=project_name&amp;i=20190105123647703gpqn26pr2&amp;t=spark&amp;id=application_1546691794888_113905562&amp;metaname=20190105123647703gpqn26pr2&amp;token=TjhlQWswZTRpYWN2L3RuK25VeE5LVy9xSUNjPSxPRFBTX09CTzoxODc1NjUzNjIyNTQzMDYxLDE1NDY5NTEwMzcseyJTdGF0ZW1lbnQiOlt7IkFjdGlvbiI6WyJvZHBzOlJlYWQiXSwiRWZmZWN0IjoiQWxsb3ciLCJSZXNvdXJjZSI6WyJhY3M6b2RwczoqOnByb2plY3RzL3FuX2JlaWppbmcvaW5zdGFuY2VzLzIwMTkwMTA1MTIzNjQ3NzAzZ3BxbjI2cHIyIl19XSwiVmVyc2lvbiI6IjEifQ==<\/p>\n<\/div>\n<ol><li class=\"MsoNormal\">\u901a\u8fc7\u65e5\u5fd7\u8f93\u51fa\u7684logview\u5728\u6d4f\u89c8\u5668\u4e2d\u53ef\u4ee5\u67e5\u770bCUPID\u7c7b\u578b\u7684\u4efb\u52a1\u6267\u884c\u7684\u57fa\u672c\u4fe1\u606f\u3002<br><img src=\"https:\/\/yqfile.alicdn.com\/8ef83335e878b349719ce76efd09428aeabba0bf.png\" width=\"300\" alt=\"8ef83335e878b349719ce76efd09428aeabba0bf\"><br><br>\n \u5355\u51fbTaskName\u4e3a<br>\n master-0\u4efb\u52a1\u6761\uff0c\u5728\u4e0b\u65b9FuxiInstance\u680f\u4e2d\uff0c\u901a\u8fc7<br>\n All\u6309\u94ae\u8fc7\u6ee4\u540e\uff0c<br><br>\n    <img src=\"https:\/\/yqfile.alicdn.com\/0661374ce536400cf6783edc1024edc93047d502.png\" width=\"300\" alt=\"0661374ce536400cf6783edc1024edc93047d502\"><br>\n <br><br>\n\u5355\u51fbTempRoot\u7684StdOut\u6309\u94ae\u53ef\u4ee5\u67e5\u770bSparkPi\u7684\u8f93\u51fa\u7ed3\u679c\uff1a<br>\n <br><br>\n    <img src=\"https:\/\/yqfile.alicdn.com\/aeae036794f80b5d112c959f4c2db33a88edc221.png\" width=\"300\" alt=\"aeae036794f80b5d112c959f4c2db33a88edc221\"><br>\n <br><br><br><\/li>\n<li class=\"MsoNormal\">\u65e5\u5fd7\u4e2d\u6253\u5370\u51fa\u4e0a\u8ff0\u7684TrackingUrl\uff0c\u8868\u793a\u60a8\u7684\u4f5c\u4e1a\u5df2\u7ecf\u63d0\u4ea4\u5230MaxCompute\u96c6\u7fa4\uff0c\u8fd9\u4e2aTrackingUrl\u975e\u5e38\u5173\u952e\uff0c\u5b83\u65e2\u662fSparkWebUI\uff0c\u4e5f\u662fHistoryServer\u7684Url\u3002\u5728\u6d4f\u89c8\u5668\u4e2d\u6253\u5f00\u8fd9\u4e2aUrl\uff0c\u53ef\u4ee5\u8ffd\u8e2aSpark\u4f5c\u4e1a\u7684\u8fd0\u884c\u60c5\u51b5\u3002<\/li>\n<\/ol><p><img src=\"https:\/\/yqfile.alicdn.com\/ec72c8a5935789fc9e23650dc9b9cda18434ad59.png\" width=\"300\" alt=\"ec72c8a5935789fc9e23650dc9b9cda18434ad59\"><br><\/p>\n<p><\/p>\n<p>\u5355\u51fbdriver\u7684stdout\u5373\u53ef\u4ee5\u67e5\u770bSpark\u4f5c\u4e1a\u7684\u8f93\u51fa\u5185\u5bb9\u3002<br><img src=\"https:\/\/yqfile.alicdn.com\/17568f04cda1ade8079281f5ef18bf193efc1c73.png\" width=\"300\" alt=\"17568f04cda1ade8079281f5ef18bf193efc1c73\"><br><span class=\"s3\" style=\"color:rgb(51,51,51);font-family:PingFangSC, 'helvetica neue', 'hiragino sans gb', arial, 'microsoft yahei ui', 'microsoft yahei', simsun, sans-serif;font-size:16px;font-weight:400;\">\u66f4\u591a\u7684\u5185\u5bb9\u89c1<\/span><span style=\"color:rgb(51,51,51);font-family:PingFangSC, 'helvetica neue', 'hiragino sans gb', arial, 'microsoft yahei ui', 'microsoft yahei', simsun, sans-serif;font-size:16px;font-weight:400;\">MaxCompute<\/span><span class=\"s3\" style=\"color:rgb(51,51,51);font-family:PingFangSC, 'helvetica neue', 'hiragino sans gb', arial, 'microsoft yahei ui', 'microsoft yahei', simsun, sans-serif;font-size:16px;font-weight:400;\">\u4ea7\u54c1\u5b98\u65b9\u5730\u5740\uff1a<a href=\"https:\/\/www.aliyun.com\/product\/odps\" rel=\"nofollow\" style=\"color:rgb(51,102,255);\"><span class=\"s5\">https:\/\/www.aliyun.com\/product\/odps<\/span><\/a><\/span><\/p>\n<p style=\"color:rgb(51,51,51);font-family:PingFangSC, 'helvetica neue', 'hiragino sans gb', arial, 'microsoft yahei ui', 'microsoft yahei', simsun, sans-serif;font-size:16px;font-weight:400;\"><span class=\"s3\">\u60f3\u4e86\u89e3\u66f4\u591a\u963f\u91cc\u5df4\u5df4\u5927\u6570\u636e\u8ba1\u7b97\u670d\u52a1<\/span>MaxCompute<span class=\"s3\">\uff0c\u53ef\u4ee5\u52a0\u5165\u793e\u7fa4\u4e00\u8d77\u4ea4\u6d41\u3002<\/span><br><\/p>\n<p style=\"color:rgb(51,51,51);font-family:PingFangSC, 'helvetica neue', 'hiragino sans gb', arial, 'microsoft yahei ui', 'microsoft yahei', simsun, sans-serif;font-size:16px;font-weight:400;\"><span class=\"s3\"><img src=\"https:\/\/yqfile.alicdn.com\/5b980f6d43bd869ad04a2786007a95e5378086b3.png\" alt=\"5b980f6d43bd869ad04a2786007a95e5378086b3\"><br><\/span><\/p>\n\n<\/div>\n<\/div>\n<div class=\"reward___3M6_P\" style=\"color:rgb(89,89,89);font-family:'Chinese Quote', '-apple-system', 'system-ui', 'Segoe UI', Roboto, 'PingFang SC', 'Hiragino Sans GB', 'Microsoft YaHei', 'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:14px;\">\n<div>\n<div class=\"lark-like lark-like-vertical\"><\/div>\n<\/div>\n<\/div>\n\n        \n        \n                \n    <\/div>          <\/div>\n                  <\/div>","orderid":"0","title":"\n &nbsp;  &nbsp;  &nbsp;  &nbsp;  &nbsp;  &nbsp;  &nbsp;  &nbsp;  &nbsp;  &nbsp;   MaxCompute Spark\u5f00\u53d1\u6307\u5357\n &nbsp;  ","smalltitle":"","mid":"0","fname":"Spark","special_id":"0","bak_id":"0","info":"0","hits":"493","pages":"1","comments":"0","posttime":"2019-05-16 01:13:27","list":"1557940407","username":"admin","author":"","copyfrom":"","copyfromurl":"","titlecolor":"","fonttype":"0","titleicon":"0","picurl":"https:\/\/www.cppentry.com\/upload_files\/","ispic":"0","yz":"1","yzer":"","yztime":"0","levels":"0","levelstime":"0","keywords":"<A HREF='https:\/\/www.cppentry.com\/do\/search.php?type=keyword&keyword=%26nbsp' target=_blank>&nbsp<\/A> <A HREF='https:\/\/www.cppentry.com\/do\/search.php?type=keyword&keyword=MaxCompute' target=_blank>MaxCompute<\/A> <A HREF='https:\/\/www.cppentry.com\/do\/search.php?type=keyword&keyword=Spark' target=_blank>Spark<\/A> <A HREF='https:\/\/www.cppentry.com\/do\/search.php?type=keyword&keyword=%BF%AA%B7%A2%D6%B8%C4%CF' target=_blank>\u5f00\u53d1\u6307\u5357<\/A>","jumpurl":"","iframeurl":"","style":"","template":"a:3:{s:4:\"foot\";s:0:\"\";s:8:\"bencandy\";s:0:\"\";s:4:\"head\";s:0:\"\";}","target":"0","ip":"47.106.78.186","lastfid":"0","money":"0","buyuser":"","passwd":"","allowdown":"","allowview":"","editer":"","edittime":"0","begintime":"0","endtime":"0","description":"                                                                                    MaxCompute Spark\u5f00\u53d1\u6307\u53570. \u6982\u8ff0\u672c\u6587\u6863\u9762\u5411\u9700\u8981\u4f7f\u7528MaxCompute Spark\u8fdb\u884c\u5f00\u53d1\u7684\u7528\u6237\u4f7f\u7528\u3002\u672c\u6307\u5357\u4e3b\u8981\u9002\u7528\u4e8e\u5177\u5907\u6709Spark\u5f00\u53d1\u7ecf\u9a8c\u7684\u5f00\u53d1\u4eba\u5458\u3002MaxCompute Spark\u662fMaxCompute\u63d0..","lastview":"1713858792","digg_num":"0","digg_time":"0","forbidcomment":"0","ifvote":"0","heart":"","htmlname":"","city_id":"0"},"page":"1"}