Skip to main content
 首页 » 编程设计

python之使用 mrjob python 将 Comomn 爬网位置作为 Amazon EMR 的输入

2025年05月04日32xxx_UU

自从我开始使用 mrjob 才几天,我已经尝试了某些中低级别的任务。现在我坚持使用 Common crawl [now onwards will be known as CC] location as使用 python mrjob 输入到 emr

我的配置文件如下所示:

runners: 
  emr: 
    aws_access_key_id: <AWS Access Key> 
    aws_secret_access_key: <AWS Secret Access Key> 
    aws_region: us-east-1 
    ec2_key_pair: cslab 
    ec2_key_pair_file: ~/cslab.pem 
    ec2_instance_type: m1.small 
    num_ec2_instances: 5 
  local: 
    base_tmp_dir: /tmp 

Big thing through small :我正在尝试获取网站网页中的字数

大事大事:下面是我的代码

我的代码:

import warc 
 
class MRcount(MRJob): 
    # ... 
 
    def mapper(self, _, s3_path): 
        s3_url_parsed = urlparse.urlparse(s3_url) 
        bucket_name = s3_url_parsed.netloc 
        key_path = s3_url_parsed.path[1:] 
        conn = boto.connect_s3() 
        bucket = conn.get_bucket('aws-publicdatasets', validate=False) 
        key = Key(bucket, s3_path) 
        webpage_text = record.payload.read() 
        yield record.header['warc-target-uri'],len(webpage_text.split() 
if __name__ == '__main__': 
    MRcount.run()) 

到目前为止一切都很好,但是当我尝试运行它时。

命令:

$ python mr_crawl.py -r emr s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2014-52/wet.paths.gz 

错误:

boto.exception.S3ResponseError: S3ResponseError: 301 Moved Permanently 
<?xml version="1.0" encoding="UTF-8"?> 
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message <RequestId>06660583263444FC</RequestId><Bucket>smarkets-db</Bucket><HostId>TCZJTKZ8wo8V1h0xjkOI6grojs/r9IBkhMOcvolXv06QEtxTX89M55aLTPGOo/ht</HostId><Endpoint>eu-west-bucket.s3.amazonaws.com</Endpoint></Error> 

我认为这是因为我的配置文件中的区域并删除了它但是我得到了一个新的错误

我的新配置文件:

runners: 
  emr: 
    aws_access_key_id: <AWS Access Key> 
    aws_secret_access_key: <AWS Secret Access Key> 
    ec2_key_pair: cslab 
    ec2_key_pair_file: ~/cslab.pem 
    ec2_instance_type: m1.small 
    num_ec2_instances: 5 
  local: 
    base_tmp_dir: /tmp 

我收到以下错误 SSH 错误:

using configs in /etc/mrjob.conf 
using existing scratch bucket mrjob-4db6342a70e021ad 
using s3://mrjob-4db6342a70e021ad/tmp/ as our scratch dir on S3 
creating tmp directory /tmp/word_count.20140603.181541.006786 
writing master bootstrap script to /tmp/word_count.20140603.181541.006786/b.py 
Copying non-input files into s3://mrjob-4db6342a70e021ad/tmp/word_count.matthew.20140603.181541.006786/files/ 
Waiting 5.0s for S3 eventual consistency 
Creating Elastic MapReduce job flow 
Job flow created with ID: j-3DCN7LULSRILW 
Created new job flow j-3DCN7LULSRILW 
Job on job flow j-3DCN7LULSRILW failed with status FAILED: The given SSH key name was invalid 
Logs are in s3://mrjob-4db6342a70e021ad/tmp/logs/j-3DCN7LULSRILW/ 
Scanning S3 logs for probable cause of failure 
Waiting 5.0s for S3 eventual consistency 
Terminating job flow: j-3DCN7LULSRILW 
Traceback (most recent call last): 
  File "word_count.py", line 16, in <module> 
    MRcount.run() 
  File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 494, in run 
    mr_job.execute() 
  File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 512, in execute 
    super(MRJob, self).execute() 
  File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 147, in execute 
    self.run_job() 
  File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 208, in run_job 
    runner.run() 
  File "/usr/local/lib/python2.7/dist-packages/mrjob/runner.py", line 458, in run 
    self._run() 
  File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 809, in _run 
    self._wait_for_job_to_complete() 
  File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 1599, in _wait_for_job_to_complete 
    raise Exception(msg) 
Exception: Job on job flow j-3DCN7LULSRILW failed with status FAILED: The given SSH key name was invalid 

谢谢,

请您参考如下方法:

在您的 MrJob 配置中,您需要根据 aws 控制台中的 key 对列表设置 ec2_key_pair

runners: 
  emr: 
    aws_access_key_id: <AWS Access Key> 
    aws_secret_access_key: <AWS Secret Access Key> 
    ec2_key_pair: cslab # <---- this name doesnt exist inside aws, so aws doesnt know the public key to use 
    ec2_key_pair_file: ~/cslab.pem # <-- you can comment this out if you dont need to login to the machine via ssh 
    ec2_instance_type: m1.small 
    num_ec2_instances: 5 
  local: 
    base_tmp_dir: /tmp 

要查看 aws 中的 key 对列表,see this doc