$ fetch http://pkg0.isc.freebsd.org/freebsd:10:x86:64/latest/Latest/pkg.txz
pkg.txz 100% of 2013 kB 552 kBps 00m03s
$
$ tar xvzf ./pkg.txz --include "*pkg-static" --strip-components 4
x pkg-static
$
（rootになって）
# ./bin/pkg-static add ./pkg.txz
Installing pkg-1.2.7_3... done
If you are upgrading from the old package format, first run:
# pkg2ng
root@hdmock:/home/doe #
$ pkg search hadoop
apache-hadoop-1.2.1_1
$ sudo pkg install hadoop
Updating repository catalogue
pkg: No packages matching 'hadoop' available in the repositories
[doe@hdmock ~]$ sudo pkg install apache-hadoop
Updating repository catalogue
The following 39 packages will be installed:
(中略)
[39/39] Installing apache-hadoop-1.2.1_1...===> Creating users and/or groups.
Creating group 'hadoop' with gid '955'.
Creating user 'hadoop' with uid '955'.
done
======================================================================
This OpenJDK implementation requires fdescfs(5) mounted on /dev/fd and
procfs(5) mounted on /proc.
If you have not done it yet, please do the following:
mount -t fdescfs fdesc /dev/fd
mount -t procfs proc /proc
To make it permanent, you need the following lines in /etc/fstab:
fdesc /dev/fd fdescfs rw 0 0
proc /proc procfs rw 0 0
=====================================================================
$ sudo mount -a
$ mount
/dev/ada0p2 on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)
fdescfs on /dev/fd (fdescfs)
procfs on /proc (procfs, local)
$ hadoop fsck /
FSCK started by hadoop from /172.29.17.159 for path / at Wed Dec 25 18:51:37 JST 2013
（略）
/var/hadoop/mapred/system/jobtracker.info: MISSING 1 blocks of total size 4 B.Status: CORRUPT
Total size: 45638364128 B
Total dirs: 5321
Total files: 13059
Total blocks (validated): 9819 (avg. block size 4647964 B)
********************************
CORRUPT FILES: 2470
MISSING BLOCKS: 2498
MISSING SIZE: 15590981155 B
CORRUPT BLOCKS: 2498
********************************
Minimally replicated blocks: 7321 (74.559525 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 7321 (74.559525 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 0.7455953
Corrupt blocks: 2498
Missing replicas: 7321 (100.0 %)
Number of data-nodes: 1
Number of racks: 1
FSCK ended at Wed Dec 25 18:51:38 JST 2013 in 1185 milliseconds
The filesystem under path '/' is CORRUPT
$ hadoop fsck / -delete
FSCK started by hadoop from /172.29.17.159 for path / at Wed Dec 25 18:58:29 JST 2013
.
/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201308300823_0001/job.jar: CORRUPT block blk_3947350403157044322
/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201308300823_0001/job.jar: MISSING 1 blocks of total size 66249 B.FSCK ended at Wed Dec 25 08:58:29 JST 2013 in 5 milliseconds
Cannot delete /tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201308300823_0001/job.jar. Name node is in safe mode.
The ratio of reported blocks 0.7456 has not reached the threshold 0.9990. Safe mode will be turned off automatically.
Fsck on path '/' FAILED
Safe mode is entered automatically at Namenode startup, and leaves safe mode automatically when the configured minimum percentage of blocks satisfies the minimum replication condition. Safe mode can also be entered manually, but then it can only be turned off manually as well.
Safe mode maintenance command. Safe mode is a Namenode state in which it
1. does not accept changes to the name space (read-only)
2. does not replicate or delete blocks.
$ hadoop fsck / -delete
FSCK started by hadoop from /172.29.17.159 for path / at Wed Dec 25 19:05:54 JST 2013
（略）
.......Status: HEALTHY
Total size: 29928657533 B
Total dirs: 5321
Total files: 10589
Total blocks (validated): 7319 (avg. block size 4089173 B)
Minimally replicated blocks: 7319 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 7319 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 1.0
Corrupt blocks: 0
Missing replicas: 7319 (100.0 %)
Number of data-nodes: 1
Number of racks: 1
FSCK ended at Wed Dec 25 19:05:55 JST 2013 in 1696 milliseconds
The filesystem under path '/' is HEALTHY
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys os subprocess
for line in sys.stdin:
line.strip("\n")
filename = line
cat = subprocess.Popen(["hadoop","dfs","-cat",filename],stdout=subprocess.PIPE)
for logline in cat.stdout:
#処理
print line
IPアドレスは、Apacheログにおいて、スペースを区切りにした第一フィールドに記載される。
だから一行ずつログを読んで、行頭のIPアドレスを抜き出し、その都度「＜IPアドレス＞＜タブ＞1」を出力する。
「if “newsyslog” not in line:」は、システムメッセージ行を読み飛ばすため。
#!/usr/bin/env python
import sys
for line in sys.stdin:
if "newsyslog" not in line:
fields = line.strip().split()
print '%s\t%s' % (fields[0],1)
$ hadoop namenode -format
Warning: $HADOOP_HOME is deprecated.
13/07/23 20:12:16 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = localhost/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.0
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1214675; compiled by 'hortonfo' on Thu Dec 15 16:36:35 UTC 2011
************************************************************/
13/07/23 20:12:18 INFO util.GSet: VM type = 64-bit
13/07/23 20:12:18 INFO util.GSet: 2% max memory = 19.33375 MB
13/07/23 20:12:18 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/07/23 20:12:18 INFO util.GSet: recommended=2097152, actual=2097152
13/07/23 20:12:19 INFO namenode.FSNamesystem: fsOwner=vanilla
13/07/23 20:12:20 INFO namenode.FSNamesystem: supergroup=supergroup
13/07/23 20:12:20 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/07/23 20:12:20 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/07/23 20:12:20 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/07/23 20:12:20 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/07/23 20:12:20 INFO common.Storage: Image file of size 113 saved in 0 seconds.
13/07/23 20:12:20 INFO common.Storage: Storage directory /tmp/hadoop-vanilla/dfs/name has been successfully formatted.
13/07/23 20:12:20 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/
