adoop是一個由Apache基金會所開發的開源分布式系統基礎架構。用戶可以在不了解分布式底層細節的情況下,開發分布式程序,充分利用集群的威力進行高速運算和存儲。Hadoop得以在大數據處理應用中廣泛應用得益于其自身在數據提取、變形和加載(ETL)方面上的天然優勢。Hadoop的分布式架構,將大數據處理引擎盡可能的靠近存儲。
為了幫助大家進一步了解Hadoop,云棲社區組織翻譯了GitHub Awesome Hadoop 資源,涵蓋Hadoop中常見的庫與工具、存儲方式、數據庫,以及相關的書籍、網站等資源。
目錄:
Awesome Hadoop
Hadoop
YARN
NoSQL
Hadoop上的SQL
數據管理
工作流、生命周期及管理
數據提取與整合
DSL
庫和工具
實時數據處理
分布式計算和編程
包裝、配置和監測
監測
搜索
安全性
基準
機器學習和大數據分析
其他
資源
網站
演示
書籍
Hadoop & 大數據事件
Hadoop
Apache Tez – 它是一個針對Hadoop數據處理應用程序的新分布式執行框架,該框架基于YARN;
SpatialHadoop – SpatialHadoop是Apache Hadoop的MapReduce擴展,專門用于處理空間數據;
GIS Tools for Hadoop –用于Hadoop框架的大數據空間分析;
Elasticsearch Hadoop – Elasticsearch與Hadoop深度集成,可用于實時搜索和分析,支持Map/Reduce、 Cascading、Apache Hive和Apache Pig;
dumbo - Python模塊,使Hadoop程序的編寫和運行更為容易;
hadoopy – 用Cython寫的Python MapReduce庫;
mrjob - mrjob是一個Python2.5+程序包,可以幫助編寫和運行Hadoop工作流;
pydoop -為Hadoop提供Python API的程序包;
hdfs-du -Hadoop分布式文件系統(HDFS)的交互可視化;
White Elephant - Hadoop的日志聚合器和儀表板;
Kiji Project
Genie - Genie提供REST-ful API,以便運行Hadoop、Hive和Pig jobs,還管理多個Hadoop資源,并在它們之間進行作業提交;
Apache Kylin –最初來自eBay公司的開源分布式分析引擎,能提供Hadoop之上的SQL查詢接口及多維分析(OLAP),以支持超大規模數據集;
Crunch -基于Go的工具包,用于在Hadoop上的ETL和特征提?。?/p>
Apache Ignite -分布式內存平臺。
YARN
Apache Slider - Apache Slider是Apache軟件基金會的孵化項目,旨在能夠輕松地實現現有應用程序到YARN集群的部署;
Apache Twill - Apache Twill是Apache Hadoop? YARN的抽象層,降低了開發分布式應用程序的復雜度,讓開發者更專注于自己的應用邏輯;
mpich2-yarn –在YARN上運行MPICH2。
NoSQL
下一代數據庫大多定位于以下幾點:非關系型、分布式、開放源碼和橫向擴展。
Apache HBase - Apache HBase;
Apache Phoenix – Hbase的SQL驅動,支持輔助索引;
happybase -一個開發者友好型的Python庫,用于Apache HBase的交互;
Hannibal –用于監測和維護HBase 集群的工具;
Haeinsa –用于HBase的線性可擴展多行多表交易庫;
hindex – Hbase的輔助索引;
Apache Accumulo - Apache Accumulo可排序分布式鍵/值存儲,是一個強大的、可擴展高性能數據存儲和檢索系統;
OpenTSDB -可擴展時間序列數據庫;
Apache Cassandra
Hadoop中的SQL
Apache Hive
Apache Phoenix - Hbase的SQL驅動,支持輔助索引;
Pivotal HAWQ – Hadoop上的并行數據庫;
Lingual -用于級聯的SQL接口(MR / TEZ工作發生器);
Cloudera Impala
Presto –用于大數據的分布式SQL查詢引擎,該查詢引擎由Facebook開發,現已開源;
Apache Tajo - Apache Hadoop的數據倉庫系統;
Apache Drill
數據管理
Apache Calcite -動態數據管理框架;
Apache Atlas -用于元數據標記及類群捕獲,支持復雜的商業數據分類。
工作流,生命周期及管理
Apache Oozie - Apache Oozie;
Azkaban
Apache Falcon -數據管理與處理平臺;
Apache NiFi -數據流系統;
AirFlow – AirFlow是以編程方式建立、調度和監控數據管道的平臺;
Luigi - Python包,用于構建批處理作業的復雜管道。
數據提取及整合
Apache Flume - Apache Flume;
Suro - Netflix分布式數據管道;
Apache Sqoop - Apache Sqoop;
Apache Kafka - Apache Kafka;
Gobblin from LinkedIn – Hadoop的通用數據提取框架;
DSL
Apache Pig - Apache Pig
Apache DataFu – Hadoop中用于處理大規模數據的庫的集合;
vahara –基于Apache Pig的機器學習和自然語言處理;
packetpig -用于開源大數據安全性分析;
akela – Mozilla的實用工具庫,用于Hadoop、HBase、Pig等等;
seqpig -Hadoop中用于大型定序數據集的簡單可擴展腳本(bioinfomation除外);
Lipstick – Pig工作流程可視化工具;A(pache)的Lipstick簡介;
PigPen - PigPen 是Clojure或分布式Clojure的Map-reduce,能夠編譯Apache Pig,但是不需要過多了解Pig也可以使用PigPen。
庫和工具
Kite Software Development Kit –一組庫、工具、示例和文檔;
gohadoop - Apache Hadoop YARN的本地Go客戶端;
Hue – 用Apache Hadoop分析數據的Web界面;
Apache Zeppelin -基于Web的筆記,可進行交互式數據分析;
Jumbune - Jumbune是為分析Hadoop集群和MapReduce作業而構建的開源產品;
Apache Thrift
Apache Avro - Apache Avro是一個數據序列化系統;
Elephant Bird – Twitter中LZO、緩沖協議相關的Hadoop、Pig、Hive和HBase代碼的集合;
Spring for Apache Hadoop
hdfs - A native go client for HDFS
Oozie Eclipse Plugin -Eclipse中用于編輯Apache Oozie工作流的圖形編輯器。
實時數據處理
Apache Storm
Apache Samza
Apache Spark
Apache Flink - Apache Flink是高效的分布式通用數據處理的平臺,用于精準的流處理。
分布式計算和編程
Apache Spark
Spark Packages - Apache Spark中程序包的community(社區)索引;
SparkHub - Apache Spark的社區;
Apache Crunch
Cascading - Cascading是在Hadoop上構建數據應用的成熟的應用開發平臺;
Apache Flink - Apache Flink是高效的分布式通用數據處理的平臺;
Apache Apex (incubating) -企業級的統一流處理和批處理引擎。
包裝,配置與監測
Apache Bigtop - 用于Apache Hadoop生態系統的包裝和測試;
Apache Ambari - Apache Ambari
Ganglia Monitoring System
ankush -一個大數據集群管理工具,用于創建和管理不同的技術集群;
Apache Zookeeper - Apache Zookeeper
Apache Curator - 用于ZooKeeper的客戶端簡化包裝和豐富ZooKeeper框架;
Buildoop - Hadoop生態系統生成器;
Deploop - Hadoop的部署系統;
Jumbune -一個用于開源MapReduce分析,MapReduce流程調試,HDFS數據質量校驗和Hadoop集群監測的工具;
inviso - Inviso是一個輕量級的工具,它提供搜索Hadoop作業,可視化性能,查看集群利用率的能力。
搜索
ElasticSearch
SenseiDB
Apache Solr -開源、分布式、實時、半結構化的數據庫;
Banana - Apache Solr的Kibana端口。
搜索引擎框架
Apache Nutch –Apache Nutch是一個高度可擴展的,可伸縮的開源網絡爬蟲軟件項目。
安全性
Apache Ranger - Ranger是一個框架,能夠跨Hadoop平臺啟用、監控和全面管理數據安全性;
Apache Sentry - Hadoop的一個授權模塊;
Apache Knox Gateway –用于與Hadoop集群交互的REST API網關。
基準
Big Data Benchmark
HiBench
Big-Bench
hive-benchmarks
hive-testbench –一個測試平臺,用于進行任何規模數據的Apache Hive實驗;
YCSB -雅虎云服務基準(YCSB)是一個開源規范和程序套件,用于評估計算機程序的檢索和維護功能;它常被用于比較NoSQL數據庫管理系統的相對性能。
機器學習和大數據分析
Apache Mahout
Oryx 2 –基于Spark、Kafka的Lambda架構,用于實時大規模的機器學習;
MLlib - MLlib是Apache Spark的可擴展機器學習庫;
R - R是用于統計計算和圖形的自由軟件環境;
RHadoop -包括RHDFS、RHBase、RMR2和plyrmr;
RHive –用于從R中開始Hive查詢;
Apache Lens
其它
Hive Plugins
1.UDF
http://nexr.github.io/hive-udf/
https://github.com/edwardcapriolo/hive_cassandra_udfs
https://github.com/livingsocial/HiveSwarm
https://github.com/ThinkBigAnalytics/Hive-Extensions-from-Think-Big-Analytics
https://github.com/karthkk/udfs
https://github.com/twitter/elephant-bird - Twitter
https://github.com/lovelysystems/ls-hive
https://github.com/stewi2/hive-udfs
https://github.com/klout/brickhouse
https://github.com/markgrover/hive-translate (PostgreSQL translate())
https://github.com/deanwampler/HiveUDFs
https://github.com/myui/hivemall (Machine Learning UDF/UDAF/UDTF)
https://github.com/edwardcapriolo/hive-geoip (GeoIP UDF)
https://github.com/Netflix/Surus
Storage Handler
https://github.com/dvasilen/Hive-Cassandra
https://github.com/yc-huang/Hive-mongo
https://github.com/balshor/gdata-storagehandler
https://github.com/karthkk/hive-hbase-json
https://github.com/sunsuk7tp/hive-hbase-integration
https://bitbucket.org/rodrigopr/redisstoragehandler
https://github.com/zhuguangbin/HiveJDBCStorageHanlder
https://github.com/chimpler/hive-solr
https://github.com/bfemiano/accumulo-hive-storage-manager
SerDe
https://github.com/rcongiu/Hive-JSON-Serde
https://github.com/mochi/hive-json-serde
https://github.com/ogrodnek/csv-serde
https://github.com/parag/HiveJsonSerde
https://github.com/johanoskarsson/hive-json-serde
https://github.com/electrum/hive-serde - JSON
https://github.com/karthkk/hive-hbase-json
Libraries and tools
https://github.com/forward3d/rbhive
https://github.com/synctree/activerecord-hive-adapter
https://github.com/hrp/sequel-hive-adapter
https://github.com/forward/node-hive
https://github.com/recruitcojp/WebHive
shib - WebUI for query engines: Hive and Presto
clive - Clojure library for interacting with Hive via Thrift
https://github.com/anjuke/hwi
https://code.google.com/a/apache-extras.org/p/hipy/
https://github.com/dmorel/Thrift-API-HiveClient2 (Perl - HiveServer2)
PyHive - Python interface to Hive and Presto
https://github.com/recruitcojp/OdbcHive
Hive-Sharp
HiveRunner - An Open Source unit test framework for hadoop hive queries based on JUnit4
Beetest - A super simple utility for testing Apache Hive scripts locally for non-Java developers.
Hive_test- Unit test framework for hive and hive-service
Flume Plugins
Flume MongoDB Sink
Flume HornetQ Channel
Flume MessagePack Source
Flume RabbitMQ source and sink
Flume UDP Source
Stratio Ingestion - Custom sinks: Cassandra, MongoDB, Stratio Streaming and JDBC
Flume Custom Serializers
Real-time analytics in Apache Flume
.Net FlumeNG Clients
資源
還有各種書籍、網站和文章等相關的資源,列表如下:
網站
有用的網站和文章
Hadoop Weekly(譯:Hadoop周刊)
The Hadoop Ecosystem Table(譯:Hadoop生態系統表)
Hadoop 1.x vs 2(譯:Hadoop的1.x vs 2)
Apache Hadoop YARN: Yet Another Resource Negotiator(Apache Hadoop YARN:另一種資源談判)
Introducing Apache Hadoop YARN(譯:Apache Hadoop YARN簡介)
Apache Hadoop YARN - Background and an Overview(譯:Apache Hadoop YARN——背景和概述)
Apache Hadoop YARN - Concepts and Applications(譯:Apache Hadoop YARN——概念與應用)
Apache Hadoop YARN - ResourceManager(譯:Apache Hadoop YARN - ResourceManager)
Apache Hadoop YARN - NodeManager(譯:Apache Hadoop YARN - NodeManager)
Migrating to MapReduce 2 on YARN (For Users)(譯:遷移到YARN上的MapReduce 2(針對用戶))
Migrating to MapReduce 2 on YARN (For Operators) (譯:遷移到YARN上的MapReduce 2(針對運營商))
Hadoop and Big Data: Use Cases at Salesforce.com(譯:Hadoop和大數據:Salesforce.com中的用例)
All you wanted to know about Hadoop, but were too afraid to ask: genealogy of elephants.(譯:你想了解卻不敢詢問的Hadoop知識:大象的家譜)
What is Bigtop, and Why Should You Care?(譯:什么是Bigtop?為什么要關注Bigtop?)
Hadoop - Distributions and Commercial Support(譯:Hadoop的分布和商業支持)
Ganglia configuration for a small Hadoop cluster and some troubleshooting(譯:小型Hadoop集群的Ganglia配置和檢修)
Hadoop illuminated - Open Source Hadoop Book(譯:Hadoop啟示——開源Hadoop書)
NoSQL Database(譯:NoSQL數據庫)
10 Best Practices for Apache Hive(譯:Apache Hive的10個最佳實踐)
Hadoop Operations at Scale
AWS BigData Blog(譯:AWS大數據博客)
Hadoop360
演示
Hadoop Summit Presentations - Slide decks from Hadoop Summit(譯:Hadoop的峰會展示)
Hadoop 24/7
An example Apache Hadoop Yarn upgrade(譯:Apache Hadoop Yarn更新實例)
Apache Hadoop In Theory And Practice(譯:Apache Hadoop的理論和實踐)
Hadoop Operations at LinkedIn(譯:Hadoop在LinkedIn的操作)
Hadoop Performance at LinkedIn(譯:Hadoop在LinkedIn的性能)
Docker based Hadoop provisioning(譯:基于Docker的Hadoop配置)
書籍
Hadoop: The Definitive Guide(譯:Hadoop權威指南)
Hadoop Operations(譯:Hadoop運營)
Apache Hadoop Yarn(譯:Apache Hadoop Yarn)
HBase: The Definitive Guide(譯:HBase權威指南)
Programming Pig(譯:Pig程序設計)
Programming Hive(譯:Hive程序設計)
Hadoop in Practice, Second Edition(譯:Hadoop實踐,第二版)
Hadoop in Action, Second Edition(譯:Hadoop實戰,第二版)
Hadoop&大數據事件
ApacheCon
Strata + Hadoop World
Hadoop Summit
以上為Hadoop學習資源集合的全部內容,更多精彩敬請期待。
若想獲取所有“史上最全Hadoop學習”資源,請關注云棲社區微信公眾號:yunqiinsight,并回復“史上最全Hadoop學習”。
在進行Flash操作時,一般我們需要設計一套Flash抽象層,至少要包括flash的init,read、write、erase這些操作。但每更換一個單片機或者flash器件就要額外去編寫flash驅動去適配init,read、write、erase。盡管有會者不難的屬性加持,但適配所有的單片機或者flash器件,工作量也可想而知。
本文為大家提供一個適配幾乎任意單片機型號的flash驅動,之所以說是幾乎,是因為我們這次要借東風,而這個東風就是Keil的FLM文件。
項目開源地址:https://gitee.com/Aladdin-Wang/flash_blob
熟悉Keil的朋友們都知道,當我們要下載編譯好的鏡像到Flash時,首先要做的一步就是選擇合適的Flash下載算法,而這個算法本身就是一個FLM文件:
所謂Flash下載算法,是負責擦除,下載應用數據到flash的一個軟件。而Keil往往會集成不少FLM文件以支持大多數的flash型號。當然,這些算法也是根據不同型號的flash所編寫的。只不過,前人們已經為我們種好了大樹,我們可以直接在樹下乘涼了。
嵌入式物聯網需要學的東西真的非常多,千萬不要學錯了路線和內容,導致工資要不上去!
無償分享大家一個資料包,差不多150多G。里面學習內容、面經、項目都比較新也比較全!某魚上買估計至少要好幾十。
點擊這里找小助理0元領?。杭游⑿蓬I取資料
Keil規定了FLM文件的構成,它是一成不變的,我們才可以放心的對文件本身進行解析,并為自己所用。
生成FLM文件的程序中,有兩個非常重要的文件,分別是
FlashPrg.c文件包含強制性flash編程函數Init、UnInit、EraseSector和ProgramPage。可選地,根據設備特性,可以實現函數EraseChip、BlankCheck和Verify 。
Function NameIndicationDescriptionBlankCheckoptionalCheck and compare patterns.EraseChipoptionalDelete entire Flash memory content.EraseSectormandatoryDelete Flash memory content of a specific sector.InitmandatoryInitialize and prepare device for Flash programming.ProgramPagemandatoryWrite the application into the Flash memory.UnInitmandatoryDe-initialize the microcontroller after one of the Flash programming steps.VerifyoptionalCompare Flash memory content with the program code.
其中有4個是必須要有的,我們來逐一說明:
int Init (unsigned long adr, unsigned long clk, unsigned long fnc);
參數adr指定設備的基址。
參數clk指定用于編程設備的時鐘頻率。
參數fnc是一個數字:
1 代表擦除。2代表程序。3代表驗證。
Code Example:
int Init (unsigned long adr, unsigned long clk, unsigned long fnc) {
// Zero Wait State
FLASH->ACR=0x00000000;
// Unlock Flash
FLASH->KEYR=FLASH_KEY1;
FLASH->KEYR=FLASH_KEY2;
// Test if IWDG is running (IWDG in HW mode)
if ((FLASH->OBR & 0x04)==0x00) {
// Set IWDG time out to ~32.768 second
IWDG->KR=0x5555; // Enable write access to IWDG_PR and IWDG_RLR
IWDG->PR=0x06; // Set prescaler to 256
IWDG->RLR=4095; // Set reload value to 4095
}
return (0);
}
int ProgramPage ( unsigned long adr, unsigned long sz, unsigned char *buf);
參數adr指定要編程的頁面的起始地址。它由主機編程系統與flash頁面的起始地址對齊。
參數sz指定數據緩沖區中的數據大小。主機編程系統確保不跨越頁面邊界。
參數buf指向包含要編程的數據的數據緩沖區。
Code Example:
int ProgramPage (unsigned long adr, unsigned long sz, unsigned char *buf) {
sz=(sz + 1) & ~1; // Adjust size for Half Words
while (sz) {
FLASH->CR |=FLASH_PG; // Programming Enabled
M16(adr)=*((unsigned short *)buf); // Program Half Word
while (FLASH->SR & FLASH_BSY);
FLASH->CR &=~FLASH_PG; // Programming Disabled
// Check for Errors
if (FLASH->SR & (FLASH_PGERR | FLASH_WRPRTERR)) {
FLASH->SR |=FLASH_PGERR | FLASH_WRPRTERR;
return (1); // Failed
}
// Go to next Half Word
adr +=2;
buf +=2;
sz -=2;
}
return (0); // Done
}
int EraseSector (unsigned long adr);
參數adr扇區地址
Code Example:
int EraseSector (unsigned long adr) {
FLASH->CR |=FLASH_PER; // Page Erase Enabled
FLASH->AR=adr; // Page Address
FLASH->CR |=FLASH_STRT; // Start Erase
while (FLASH->SR & FLASH_BSY) {
IWDG->KR=0xAAAA; // Reload IWDG
}
FLASH->CR &=~FLASH_PER; // Page Erase Disabled
return (0); // Done
}
int UnInit (unsigned long fnc);
Code Example
int UnInit (unsigned long fnc) {
// Lock Flash
FLASH->CR |=FLASH_LOCK;
return (0);
}
文件FlashDev.c包含以下參數定義:
struct FlashDevice const FlashDevice={
FLASH_DRV_VERS, // 驅動版本,請勿修改!
"New Device 256kB Flash" , // 設備名稱
ONCHIP, // 設備類型
0x00000000, // 設備起始地址
0x00040000, // 以字節為單位的設備大小 (256kB)
1024, // 編程頁面大小
0, // 保留,必須為0
0xFF, // 已擦除內存的初始內容
100, // 程序頁面超時 100 毫秒
3000, // 擦除扇區超時 3000 毫秒
// 指定扇區的大小和地址
0x002000, 0x000000, // 扇區大小 8kB(8 個扇區)
0x010000, 0x010000, // 扇區大小 64kB(2 個扇區)
0x002000, 0x030000, // 扇區大小 8kB(8 個扇區)
SECTOR_END
};
Device Name通常顯示在工具中,用于識別 Flash 算法。確保此名稱反映設備名稱。編程頁面大小指定使用函數ProgramPage進行編程 的塊大小。對于塊大小較小的設備,最好指定物理塊大小的倍數,因為這可以減少與目標的通信開銷??焖倬幊痰淖罴褖K大小為 1024 字節,但系統本身并不限制此大小值。
下面讓我們解析一下現有的FLM文件,以STM32F4xx_1024.FLM為例:將ARM:CMSIS Pack文件夾(通常在D:\Users\Administrator\AppData\Local\Arm\Packs\Keil\STM32F4xx_DFP.15.0\CMSIS\Flash)中的內容復制到一個新文件夾中。
打開命令行工具,輸入arm-none-eabi-readelf -a STM32F4xx_1024.FLM:
$ arm-none-eabi-readelf -a STM32F4xx_1024.FLM
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: ARM
Version: 0x1
Entry point address: 0x0
Start of program headers: 12172 (bytes into file)
Start of section headers: 12236 (bytes into file)
Flags: 0x5000000, Version5 EABI
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 2
Size of section headers: 40 (bytes)
Number of section headers: 16
Section header string table index: 15
Section Headers:
[Nr] Name Type Addr Off Size ES Flg Lk Inf Al
[ 0] NULL 00000000 000000 000000 00 0 0 0
[ 1] PrgCode PROGBITS 00000000 000034 000144 00 AX 0 0 4
[ 2] PrgData PROGBITS 00000144 000178 000004 00 WA 0 0 4
[ 3] DevDscr PROGBITS 00000148 00017c 0010a0 00 A 0 0 4
[ 4] .debug_abbrev PROGBITS 00000000 00121c 0005a4 00 0 0 1
[ 5] .debug_frame PROGBITS 00000000 0017c0 000104 00 0 0 1
[ 6] .debug_info PROGBITS 00000000 0018c4 00064c 00 0 0 1
[ 7] .debug_line PROGBITS 00000000 001f10 000218 00 0 0 1
[ 8] .debug_loc PROGBITS 00000000 002128 0001b8 00 0 0 1
[ 9] .debug_macinfo PROGBITS 00000000 0022e0 000614 00 0 0 1
[10] .debug_pubnames PROGBITS 00000000 0028f4 000096 00 0 0 1
[11] .symtab SYMTAB 00000000 00298c 000110 10 12 9 4
[12] .strtab STRTAB 00000000 002a9c 000100 00 0 0 1
[13] .note NOTE 00000000 002b9c 00001c 00 0 0 4
[14] .comment PROGBITS 00000000 002bb8 000334 00 0 0 1
[15] .shstrtab STRTAB 00000000 002eec 0000a0 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
y (purecode), p (processor specific)
There are no section groups in this file.
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
LOAD 0x000034 0x00000000 0x00000000 0x00148 0x00148 RWE 0x4
LOAD 0x00017c 0x00000148 0x00000148 0x010a0 0x010a0 R 0x4
Section to Segment mapping:
Segment Sections...
00 PrgCode PrgData
01 DevDscr
There is no dynamic section in this file.
There are no relocations in this file.
There are no unwind sections in this file.
Symbol table '.symtab' contains 17 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 00000000 0 NOTYPE LOCAL DEFAULT UND
1: 00000000 0 NOTYPE LOCAL DEFAULT 1 $t
2: 00000122 0 NOTYPE LOCAL DEFAULT 1 $d
3: 00000144 0 NOTYPE LOCAL DEFAULT 2 $d.realdata
4: 00000148 0 NOTYPE LOCAL DEFAULT 3 $d.realdata
5: 00000000 0 FILE LOCAL DEFAULT ABS FlashPrg.c
6: 00000000 0 SECTION LOCAL DEFAULT 1 .text
7: 00000000 0 FILE LOCAL DEFAULT ABS FlashDev.c
8: 00000148 4256 SECTION LOCAL DEFAULT 3 .constdata
9: 00000000 0 NOTYPE GLOBAL HIDDEN ABS BuildAttributes$$THM_ISAv
10: 00000001 28 FUNC GLOBAL HIDDEN 1 GetSecNum
11: 0000001d 46 FUNC GLOBAL HIDDEN 1 Init
12: 0000004b 14 FUNC GLOBAL HIDDEN 1 UnInit
13: 00000059 44 FUNC GLOBAL HIDDEN 1 EraseChip
14: 00000085 76 FUNC GLOBAL HIDDEN 1 EraseSector
15: 000000d1 82 FUNC GLOBAL HIDDEN 1 ProgramPage
16: 00000148 4256 OBJECT GLOBAL HIDDEN 3 FlashDevice
No version information found in this file.
Displaying notes found at file offset 0x00002b9c with length 0x0000001c:
Owner Data size Description
ARM 0x0000000c Unknown note type: (0x40000000)
通過Symbol table信息我們可以找到Init、UnInit、EraseSector和ProgramPage函數所在的位置。
我們還需要根據Section Headers所描述的位置提取出Prgcode(代碼),PrgData(數據),DevDscr(設備描述)的信息。
在命令行中輸入arm-none-eabi-objdum -s -d STM32F4xx_1024.FLM:-s參數可以將所有段的內容一十六進制方式打印出來,-d參數可以將所有包含指令的段反匯編。
$ arm-none-eabi-objdump -s -d STM32F4xx_1024.FLM
STM32F4xx_1024.FLM: file format elf32-littlearm
Contents of section PrgCode:
0000 0003000e 202802d3 4009001d 70471028 .... (..@...pG.(
0010 02d30009 c01c7047 80087047 42484149 ......pG..pGBHAI
0020 41604249 41600021 0160c168 f0221143 A`BIA`.!.`.h.".C
0030 c1604069 800606d4 3e483d49 01600621 .`@i....>H=I.`.!
0040 41603d49 81600020 70473748 01694205 A`=I.`. pG7H.iB.
0050 11430161 00207047 10b53348 01690424 .C.a. pG..3H.i.$
0060 21430161 0169a203 11430161 3349314a !C.a.i...C.a3I1J
0070 00e01160 c368db03 fbd40169 a1430161 ...`.h.....i.C.a
0080 002010bd 30b5fff7 bbff2749 ca68f023 . ..0.....'I.h.#
0090 1a43ca60 02240c61 0a690007 400e0243 .C.`.$.a.i..@..C
00a0 0a610869 e2031043 08612448 214a00e0 .a.i...C.a$H!J..
00b0 1060cd68 ed03fbd4 0869a043 0861c868 .`.h.....i.C.a.h
00c0 0006000f 03d0c868 1843c860 012030bd .......h.C.`. 0.
00d0 70b5154d c91c8908 eb688900 f0263343 p..M.....h...&3C
00e0 eb600023 2b61164b 17e02c69 1c432c61 .`.#+a.K..,i.C,a
00f0 14680460 ec68e403 fcd42c69 64086400 .h.`.h....,id.d.
0100 2c61ec68 2406240f 04d0e868 3043e860 ,a.h$.$....h0C.`
0110 012070bd 001d121d 091f0029 e5d10020 . p........)...
0120 70bd0000 23016745 003c0240 ab89efcd p...#.gE.<.@....
0130 55550000 00300040 ff0f0000 aaaa0000 UU...0.@........
0140 01020000 ....
Contents of section PrgData:
0144 00000000 ....
Contents of section DevDscr:
0148 01015354 4d333246 34787820 466c6173 ..STM32F4xx Flas
0158 68000000 00000000 00000000 00000000 h...............
0168 00000000 00000000 00000000 00000000 ................
0178 00000000 00000000 00000000 00000000 ................
0188 00000000 00000000 00000000 00000000 ................
0198 00000000 00000000 00000000 00000000 ................
01a8 00000000 00000000 00000000 00000000 ................
01b8 00000000 00000000 00000000 00000000 ................
01c8 00000100 00000008 00001000 00040000 ................
01d8 00000000 ff000000 64000000 70170000 ........d...p...
01e8 00400000 00000000 00000100 00000100 .@..............
01f8 00000200 00000200 ffffffff ffffffff ................
我們所需要的正是以上信息,接下來的任務只需要寫一個上位機,將以上文件提取出來即可,這個工具我已經寫好,如圖:
選擇STM32F4xx_1024.FLM,生成STM32F4xx_1024.FLM.c文件,然后直接添加到我們的工程中即可,生成的代碼如下:
#include "flash_blob.h"
#define OPS_OFFSET ((uint32_t)&flash_code)
#define DEV_OFFSET ((uint32_t)&flash_dev)
#define RAM_OFFSET ((uint32_t)&rw_data)
static const uint32_t flash_code[]={
0X0E000300,0XD3022820,0X1D000940,0X28104770,0X0900D302,0X47701CC0,0X47700880,0X49414842,
0X49426041,0X21036041,0X68C16001,0X431122F0,0X694060C1,0XD4060680,0X493D483E,0X21066001,
0X493D6041,0X20006081,0X48374770,0X05426901,0X61014311,0X47702000,0X4833B510,0X24046901,
0X61014321,0X03A26901,0X61014311,0X4A314933,0X6011E000,0X03DB68C3,0X6901D4FB,0X610143A1,
0XBD102000,0XF7FFB530,0X4927FFBB,0X23F068CA,0X60CA431A,0X610C2402,0X0700690A,0X43020E40,
0X6908610A,0X431003E2,0X48246108,0XE0004A21,0X68CD6010,0XD4FB03ED,0X43A06908,0X68C86108,
0X0F000600,0X68C8D003,0X60C84318,0XBD302001,0X4D15B570,0X08891CC9,0X008968EB,0X433326F0,
0X230060EB,0X4B16612B,0X692CE017,0X612C431C,0X60046814,0X03E468EC,0X692CD4FC,0X00640864,
0X68EC612C,0X0F240624,0X68E8D004,0X60E84330,0XBD702001,0X1D121D00,0X29001F09,0X2000D1E5,
0X0000BD70,0X45670123,0X40023C00,0XCDEF89AB,0X00005555,0X40003000,0X00000FFF,0X0000AAAA,
0X00000201,0X00000000,
};
static const uint32_t flash_dev[]={
0X54530101,0X4632334D,0X20787834,0X73616C46,0X00000068,0X00000000,0X00000000,0X00000000,
0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,
0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,
0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,0X00000000,
0X00010000,0X08000000,0X00100000,0X00000400,0X00000000,0X000000FF,0X00000064,0X00001770,
0X00004000,0X00000000,0X00010000,0X00010000,0X00020000,0X00020000,0XFFFFFFFF,0XFFFFFFFF,
};
static uint32_t rw_data[]={
0X00000000,
};
static flash_blob_device_t flash_device={
(void*)(OPS_OFFSET + 0X001D), // Init
(void*)(OPS_OFFSET + 0X004B), // UnInit
(void*)(OPS_OFFSET + 0X0059), // EraseChip
(void*)(OPS_OFFSET + 0X0085), // EraseSector
(void*)(OPS_OFFSET + 0X00D1), // ProgramPage
(void*)(DEV_OFFSET),
(void*)(RAM_OFFSET),
};
static int flash_blob_device_register(void)
{
flash_dev_register(&flash_device);
return 0 ;
}
INIT_BOARD_EXPORT(flash_blob_device_register);
接下來,為了方便后續使用,需要設計一個flash驅動抽象層,代碼如下:flash_blob.h
#ifndef FLASH_BLOB_H
#define FLASH_BLOB_H
#include "rtthread.h"
#include <stdint.h>
#include <stddef.h>
#include <stdbool.h>
#define VERS 1 // Interface Version 1.01
#define UNKNOWN 0 // Unknown
#define ONCHIP 1 // On-chip Flash Memory
#define EXT8BIT 2 // External Flash Device on 8-bit Bus
#define EXT16BIT 3 // External Flash Device on 16-bit Bus
#define EXT32BIT 4 // External Flash Device on 32-bit Bus
#define EXTSPI 5 // External Flash Device on SPI
#define SECTOR_NUM 16 // Max Number of Sector Items
struct FlashSectors {
unsigned long szSector; // Sector Size in Bytes
unsigned long AddrSector; // Address of Sector
};
typedef struct FlashDevice {
unsigned short Vers; // Version Number and Architecture
char DevName[128]; // Device Name and Description
unsigned short DevType; // Device Type: ONCHIP, EXT8BIT, EXT16BIT, ...
unsigned long DevAdr; // Default Device Start Address
unsigned long szDev; // Total Size of Device
unsigned long szPage; // Programming Page Size
unsigned long Res; // Reserved for future Extension
unsigned char valEmpty; // Content of Erased Memory
unsigned long toProg; // Time Out of Program Page Function
unsigned long toErase; // Time Out of Erase Sector Function
struct FlashSectors sectors[SECTOR_NUM];
}flash_dev_t;
typedef struct {
int (*Init)(uint32_t adr, uint32_t clk, uint32_t fnc);
int (*UnInit)(uint32_t fnc);
int (*EraseChip)(void);
int (*EraseSector)(uint32_t adr);
int (*ProgramPage)(uint32_t adr, uint32_t sz, uint8_t* buf);
}flash_ops_t;
typedef struct
{
flash_ops_t tFlashops;
flash_dev_t *ptFlashDev;
int *pPrgData;
rt_slist_t slist;
}flash_blob_t;
extern void flash_dev_register(flash_blob_t *ptFlashDevice);
extern bool target_flash_init(uint32_t flash_start, int32_t size);
extern bool target_flash_uninit(uint32_t flash_start);
extern int32_t target_flash_write(uint32_t addr, const uint8_t *buf, int32_t size);
extern int32_t target_flash_erase(uint32_t addr, int32_t size);
extern int32_t target_flash_read(uint32_t addr, const uint8_t *buf, int32_t size);
#endif
flash_blob.c
#include "flash_blob.h"
#include "stdio.h"
#include "stdlib.h"
#include "string.h"
static bool s_bIsInit=false;
static rt_slist_t _slist_head=RT_SLIST_OBJECT_INIT(_slist_head);
register int *pPrgDataBase __asm("r9");
void flash_dev_register(flash_blob_t *ptFlashDevice)
{
rt_slist_init(&(ptFlashDevice->slist));
rt_slist_append(&_slist_head, &(ptFlashDevice->slist));
}
static flash_blob_t * flash_dev_find(uint32_t flash_start)
{
rt_slist_t *node;
rt_slist_for_each(node, &_slist_head) {
flash_blob_t *ptFlashDevice=rt_slist_entry(node, flash_blob_t, slist);
if(flash_start >=ptFlashDevice->ptFlashDev->DevAdr &&
flash_start < ptFlashDevice->ptFlashDev->DevAdr + ptFlashDevice->ptFlashDev->szDev) {
return ptFlashDevice;
}
}
return NULL;
}
bool target_flash_init(uint32_t flash_start, int32_t size)
{
if (flash_start % 4 !=0) {
LOG_E("flash addr must be 4-byte alignment");
return NULL;
}
flash_blob_t *ptFlashDevice=flash_dev_find(flash_start);
if(ptFlashDevice !=NULL) {
pPrgDataBase=ptFlashDevice->pPrgData;
ptFlashDevice->tFlashops.Init(flash_start, 0, 0);
return true;
}
return false;
}
bool target_flash_uninit(uint32_t flash_start)
{
flash_blob_t *ptFlashDevice=flash_dev_find(flash_start);
if(ptFlashDevice !=NULL) {
pPrgDataBase=ptFlashDevice->pPrgData;
ptFlashDevice->tFlashops.UnInit(flash_start);
return true;
}
return true;
}
int target_flash_write(uint32_t addr, const uint8_t *buf, int32_t size)
{
flash_blob_t *ptFlashDevice=flash_dev_find(addr);
if(ptFlashDevice !=NULL) {
pPrgDataBase=ptFlashDevice->pPrgData;
while(size > 0) {
uint32_t write_size=size > ptFlashDevice->ptFlashDev->szPage ? ptFlashDevice->ptFlashDev->szPage : size;
if( 0 !=ptFlashDevice->tFlashops.ProgramPage(addr, write_size, (uint8_t *)buf)) {
LOG_E("Programming Failed");
return -1;
}
addr +=write_size;
buf +=write_size;
size -=write_size;
}
return size;
}
return -1;
}
int32_t target_flash_read(uint32_t addr, const uint8_t *buf, int32_t size)
{
flash_blob_t *ptFlashDevice;
pPrgDataBase=ptFlashDevice->pPrgData;
return size;
}
int32_t target_flash_erase(uint32_t addr, int32_t size)
{
int32_t wSector, wRemainLen;
flash_blob_t *ptFlashDevice=flash_dev_find(addr);
if(ptFlashDevice !=NULL) {
if (size > ptFlashDevice->ptFlashDev->szDev) {
LOG_E("erase outrange flash size! addr is (0x%p)\n", (void *)(addr + size));
return -1;
}
pPrgDataBase=ptFlashDevice->pPrgData;
wRemainLen=size;
while(wRemainLen > 0) {
if(0 !=ptFlashDevice->tFlashops.EraseSector(addr)) {
LOG_E("erase Failed! addr is (0x%p)\n", (void *)addr);
return -1;
}
for(wSector=0; wSector < SECTOR_NUM; wSector++) {
if(ptFlashDevice->ptFlashDev->sectors[wSector + 1].szSector==0XFFFFFFFF )
break;
if(((addr - ptFlashDevice->ptFlashDev->DevAdr) < ptFlashDevice->ptFlashDev->sectors[wSector + 1].AddrSector) &&
((addr - ptFlashDevice->ptFlashDev->DevAdr) >=ptFlashDevice->ptFlashDev->sectors[wSector].AddrSector) )
break;
}
addr +=ptFlashDevice->ptFlashDev->sectors[wSector].szSector;
wRemainLen -=ptFlashDevice->ptFlashDev->sectors[wSector].szSector;
}
return size;
}
return -1;
}
int32_t target_flash_verify (uint32_t addr, uint8_t *buf, int32_t size)
{
return size;
}
本項目借用了rtthread的自動初始化機制和鏈表,所以最快的使用方式是直接作為rtthread的軟件包使用,使用方法如下:
以上步驟完成后,就可以快速使用了,例如將YMODEM接收到的數據,寫到flash中,代碼如下:
uint8_t *ymodem_call_back_receive(uint8_t *pchBuffer, uint16_t hwSize)
{
static char *s_pchFileName=NULL, *s_pchFileSize=NULL;
static uint32_t s_wFileSize=0, s_wRemainLen=0, s_wOffSet=0;
static enum {
START=0,
RECEIVE,
END,
} s_tState={START};
switch(s_tState) {
case START:
s_wOffSet=0;
s_pchFileName=(char *)&pchBuffer[0];
s_pchFileSize=(char *)&pchBuffer[strlen(s_pchFileName) + 1];
s_wFileSize=atol(s_pchFileSize);
LOG_D("Ymodem file_name:%s", s_pchFileName);
LOG_D("Ymodem file_size:%d", s_wFileSize);
if(target_flash_init(APP_PART_ADDR, s_wFileSize)==false) {
LOG_E("target flash uninit.");
return NULL;
}
if(target_flash_erase(APP_PART_ADDR, s_wFileSize) < 0) {
LOG_E("target flash erase error.");
return NULL;
}
s_tState=RECEIVE;
break;
case RECEIVE:
s_wRemainLen=s_wFileSize - s_wOffSet;
if(hwSize > s_wRemainLen) {
hwSize=s_wRemainLen;
s_tState=END;
}
if(target_flash_write(APP_PART_ADDR + s_wOffSet, pchBuffer, hwSize) < 0) {
LOG_E("target flash write data error.");
return NULL;
}
s_wOffSet +=hwSize;
break;
case END:
target_flash_uninit(APP_PART_ADDR);
s_tState=START;
break;
}
return s_chBuffer;
}
本文介紹了如何利用Keil中FLM文件制作通用的flash驅動,但是總會遇到keil中沒有的外部flash芯片,下篇將介紹如何從零編寫生成FLM文件的工程,敬請期待...
轉載自:AIoT開源項目分享
文章來源于利用MDK的FLM文件制作通用flash驅動原文鏈接:https://mp.weixin.qq.com/s/wi2PBfiAqqt6N_f-znAOEQ
adoop是一個由Apache基金會所開發的開源分布式系統基礎架構。用戶可以在不了解分布式底層細節的情況下,開發分布式程序,充分利用集群的威力進行高速運算和存儲。Hadoop得以在大數據處理應用中廣泛應用得益于其自身在數據提取、變形和加載(ETL)方面上的天然優勢。Hadoop的分布式架構,將大數據處理引擎盡可能的靠近存儲。
為了幫助大家進一步了解Hadoop,云棲社區組織翻譯了GitHub Awesome Hadoop 資源,涵蓋Hadoop中常見的庫與工具、存儲方式、數據庫,以及相關的書籍、網站等資源。
目錄:
Awesome Hadoop
Hadoop
YARN
NoSQL
Hadoop上的SQL
數據管理
工作流、生命周期及管理
數據提取與整合
DSL
庫和工具
實時數據處理
分布式計算和編程
包裝、配置和監測
監測
搜索
安全性
基準
機器學習和大數據分析
其他
資源
網站
演示
書籍
Hadoop & 大數據事件
Hadoop
Apache Tez – 它是一個針對Hadoop數據處理應用程序的新分布式執行框架,該框架基于YARN;
SpatialHadoop – SpatialHadoop是Apache Hadoop的MapReduce擴展,專門用于處理空間數據;
GIS Tools for Hadoop –用于Hadoop框架的大數據空間分析;
Elasticsearch Hadoop – Elasticsearch與Hadoop深度集成,可用于實時搜索和分析,支持Map/Reduce、 Cascading、Apache Hive和Apache Pig;
dumbo - Python模塊,使Hadoop程序的編寫和運行更為容易;
hadoopy – 用Cython寫的Python MapReduce庫;
mrjob - mrjob是一個Python2.5+程序包,可以幫助編寫和運行Hadoop工作流;
pydoop -為Hadoop提供Python API的程序包;
hdfs-du -Hadoop分布式文件系統(HDFS)的交互可視化;
White Elephant - Hadoop的日志聚合器和儀表板;
Kiji Project
Genie - Genie提供REST-ful API,以便運行Hadoop、Hive和Pig jobs,還管理多個Hadoop資源,并在它們之間進行作業提交;
Apache Kylin –最初來自eBay公司的開源分布式分析引擎,能提供Hadoop之上的SQL查詢接口及多維分析(OLAP),以支持超大規模數據集;
Crunch -基于Go的工具包,用于在Hadoop上的ETL和特征提取;
Apache Ignite -分布式內存平臺。
YARN
Apache Slider - Apache Slider是Apache軟件基金會的孵化項目,旨在能夠輕松地實現現有應用程序到YARN集群的部署;
Apache Twill - Apache Twill是Apache Hadoop? YARN的抽象層,降低了開發分布式應用程序的復雜度,讓開發者更專注于自己的應用邏輯;
mpich2-yarn –在YARN上運行MPICH2。
NoSQL
下一代數據庫大多定位于以下幾點:非關系型、分布式、開放源碼和橫向擴展。
Apache HBase - Apache HBase;
Apache Phoenix – Hbase的SQL驅動,支持輔助索引;
happybase -一個開發者友好型的Python庫,用于Apache HBase的交互;
Hannibal –用于監測和維護HBase 集群的工具;
Haeinsa –用于HBase的線性可擴展多行多表交易庫;
hindex – Hbase的輔助索引;
Apache Accumulo - Apache Accumulo可排序分布式鍵/值存儲,是一個強大的、可擴展高性能數據存儲和檢索系統;
OpenTSDB -可擴展時間序列數據庫;
Apache Cassandra
Hadoop中的SQL
Apache Hive
Apache Phoenix - Hbase的SQL驅動,支持輔助索引;
Pivotal HAWQ – Hadoop上的并行數據庫;
Lingual -用于級聯的SQL接口(MR / TEZ工作發生器);
Cloudera Impala
Presto –用于大數據的分布式SQL查詢引擎,該查詢引擎由Facebook開發,現已開源;
Apache Tajo - Apache Hadoop的數據倉庫系統;
Apache Drill
數據管理
Apache Calcite -動態數據管理框架;
Apache Atlas -用于元數據標記及類群捕獲,支持復雜的商業數據分類。
工作流,生命周期及管理
Apache Oozie - Apache Oozie;
Azkaban
Apache Falcon -數據管理與處理平臺;
Apache NiFi -數據流系統;
AirFlow – AirFlow是以編程方式建立、調度和監控數據管道的平臺;
Luigi - Python包,用于構建批處理作業的復雜管道。
數據提取及整合
Apache Flume - Apache Flume;
Suro - Netflix分布式數據管道;
Apache Sqoop - Apache Sqoop;
Apache Kafka - Apache Kafka;
Gobblin from LinkedIn – Hadoop的通用數據提取框架;
DSL
Apache Pig - Apache Pig
Apache DataFu – Hadoop中用于處理大規模數據的庫的集合;
vahara –基于Apache Pig的機器學習和自然語言處理;
packetpig -用于開源大數據安全性分析;
akela – Mozilla的實用工具庫,用于Hadoop、HBase、Pig等等;
seqpig -Hadoop中用于大型定序數據集的簡單可擴展腳本(bioinfomation除外);
Lipstick – Pig工作流程可視化工具;A(pache)的Lipstick簡介;
PigPen - PigPen 是Clojure或分布式Clojure的Map-reduce,能夠編譯Apache Pig,但是不需要過多了解Pig也可以使用PigPen。
庫和工具
Kite Software Development Kit –一組庫、工具、示例和文檔;
gohadoop - Apache Hadoop YARN的本地Go客戶端;
Hue – 用Apache Hadoop分析數據的Web界面;
Apache Zeppelin -基于Web的筆記,可進行交互式數據分析;
Jumbune - Jumbune是為分析Hadoop集群和MapReduce作業而構建的開源產品;
Apache Thrift
Apache Avro - Apache Avro是一個數據序列化系統;
Elephant Bird – Twitter中LZO、緩沖協議相關的Hadoop、Pig、Hive和HBase代碼的集合;
Spring for Apache Hadoop
hdfs - A native go client for HDFS
Oozie Eclipse Plugin -Eclipse中用于編輯Apache Oozie工作流的圖形編輯器。
實時數據處理
Apache Storm
Apache Samza
Apache Spark
Apache Flink - Apache Flink是高效的分布式通用數據處理的平臺,用于精準的流處理。
分布式計算和編程
Apache Spark
Spark Packages - Apache Spark中程序包的community(社區)索引;
SparkHub - Apache Spark的社區;
Apache Crunch
Cascading - Cascading是在Hadoop上構建數據應用的成熟的應用開發平臺;
Apache Flink - Apache Flink是高效的分布式通用數據處理的平臺;
Apache Apex (incubating) -企業級的統一流處理和批處理引擎。
包裝,配置與監測
Apache Bigtop - 用于Apache Hadoop生態系統的包裝和測試;
Apache Ambari - Apache Ambari
Ganglia Monitoring System
ankush -一個大數據集群管理工具,用于創建和管理不同的技術集群;
Apache Zookeeper - Apache Zookeeper
Apache Curator - 用于ZooKeeper的客戶端簡化包裝和豐富ZooKeeper框架;
Buildoop - Hadoop生態系統生成器;
Deploop - Hadoop的部署系統;
Jumbune -一個用于開源MapReduce分析,MapReduce流程調試,HDFS數據質量校驗和Hadoop集群監測的工具;
inviso - Inviso是一個輕量級的工具,它提供搜索Hadoop作業,可視化性能,查看集群利用率的能力。
搜索
ElasticSearch
SenseiDB
Apache Solr -開源、分布式、實時、半結構化的數據庫;
Banana - Apache Solr的Kibana端口。
搜索引擎框架
Apache Nutch –Apache Nutch是一個高度可擴展的,可伸縮的開源網絡爬蟲軟件項目。
安全性
Apache Ranger - Ranger是一個框架,能夠跨Hadoop平臺啟用、監控和全面管理數據安全性;
Apache Sentry - Hadoop的一個授權模塊;
Apache Knox Gateway –用于與Hadoop集群交互的REST API網關。
基準
Big Data Benchmark
HiBench
Big-Bench
hive-benchmarks
hive-testbench –一個測試平臺,用于進行任何規模數據的Apache Hive實驗;
YCSB -雅虎云服務基準(YCSB)是一個開源規范和程序套件,用于評估計算機程序的檢索和維護功能;它常被用于比較NoSQL數據庫管理系統的相對性能。
機器學習和大數據分析
Apache Mahout
Oryx 2 –基于Spark、Kafka的Lambda架構,用于實時大規模的機器學習;
MLlib - MLlib是Apache Spark的可擴展機器學習庫;
R - R是用于統計計算和圖形的自由軟件環境;
RHadoop -包括RHDFS、RHBase、RMR2和plyrmr;
RHive –用于從R中開始Hive查詢;
Apache Lens
其它
Hive Plugins
1.UDF
http://nexr.github.io/hive-udf/
https://github.com/edwardcapriolo/hive_cassandra_udfs
https://github.com/livingsocial/HiveSwarm
https://github.com/ThinkBigAnalytics/Hive-Extensions-from-Think-Big-Analytics
https://github.com/karthkk/udfs
https://github.com/twitter/elephant-bird - Twitter
https://github.com/lovelysystems/ls-hive
https://github.com/stewi2/hive-udfs
https://github.com/klout/brickhouse
https://github.com/markgrover/hive-translate (PostgreSQL translate())
https://github.com/deanwampler/HiveUDFs
https://github.com/myui/hivemall (Machine Learning UDF/UDAF/UDTF)
https://github.com/edwardcapriolo/hive-geoip (GeoIP UDF)
https://github.com/Netflix/Surus
Storage Handler
https://github.com/dvasilen/Hive-Cassandra
https://github.com/yc-huang/Hive-mongo
https://github.com/balshor/gdata-storagehandler
https://github.com/karthkk/hive-hbase-json
https://github.com/sunsuk7tp/hive-hbase-integration
https://bitbucket.org/rodrigopr/redisstoragehandler
https://github.com/zhuguangbin/HiveJDBCStorageHanlder
https://github.com/chimpler/hive-solr
https://github.com/bfemiano/accumulo-hive-storage-manager
SerDe
https://github.com/rcongiu/Hive-JSON-Serde
https://github.com/mochi/hive-json-serde
https://github.com/ogrodnek/csv-serde
https://github.com/parag/HiveJsonSerde
https://github.com/johanoskarsson/hive-json-serde
https://github.com/electrum/hive-serde - JSON
https://github.com/karthkk/hive-hbase-json
Libraries and tools
https://github.com/forward3d/rbhive
https://github.com/synctree/activerecord-hive-adapter
https://github.com/hrp/sequel-hive-adapter
https://github.com/forward/node-hive
https://github.com/recruitcojp/WebHive
shib - WebUI for query engines: Hive and Presto
clive - Clojure library for interacting with Hive via Thrift
https://github.com/anjuke/hwi
https://code.google.com/a/apache-extras.org/p/hipy/
https://github.com/dmorel/Thrift-API-HiveClient2 (Perl - HiveServer2)
PyHive - Python interface to Hive and Presto
https://github.com/recruitcojp/OdbcHive
Hive-Sharp
HiveRunner - An Open Source unit test framework for hadoop hive queries based on JUnit4
Beetest - A super simple utility for testing Apache Hive scripts locally for non-Java developers.
Hive_test- Unit test framework for hive and hive-service
Flume Plugins
Flume MongoDB Sink
Flume HornetQ Channel
Flume MessagePack Source
Flume RabbitMQ source and sink
Flume UDP Source
Stratio Ingestion - Custom sinks: Cassandra, MongoDB, Stratio Streaming and JDBC
Flume Custom Serializers
Real-time analytics in Apache Flume
.Net FlumeNG Clients
資源
還有各種書籍、網站和文章等相關的資源,列表如下:
網站
有用的網站和文章
Hadoop Weekly(譯:Hadoop周刊)
The Hadoop Ecosystem Table(譯:Hadoop生態系統表)
Hadoop 1.x vs 2(譯:Hadoop的1.x vs 2)
Apache Hadoop YARN: Yet Another Resource Negotiator(Apache Hadoop YARN:另一種資源談判)
Introducing Apache Hadoop YARN(譯:Apache Hadoop YARN簡介)
Apache Hadoop YARN - Background and an Overview(譯:Apache Hadoop YARN——背景和概述)
Apache Hadoop YARN - Concepts and Applications(譯:Apache Hadoop YARN——概念與應用)
Apache Hadoop YARN - ResourceManager(譯:Apache Hadoop YARN - ResourceManager)
Apache Hadoop YARN - NodeManager(譯:Apache Hadoop YARN - NodeManager)
Migrating to MapReduce 2 on YARN (For Users)(譯:遷移到YARN上的MapReduce 2(針對用戶))
Migrating to MapReduce 2 on YARN (For Operators) (譯:遷移到YARN上的MapReduce 2(針對運營商))
Hadoop and Big Data: Use Cases at Salesforce.com(譯:Hadoop和大數據:Salesforce.com中的用例)
All you wanted to know about Hadoop, but were too afraid to ask: genealogy of elephants.(譯:你想了解卻不敢詢問的Hadoop知識:大象的家譜)
What is Bigtop, and Why Should You Care?(譯:什么是Bigtop?為什么要關注Bigtop?)
Hadoop - Distributions and Commercial Support(譯:Hadoop的分布和商業支持)
Ganglia configuration for a small Hadoop cluster and some troubleshooting(譯:小型Hadoop集群的Ganglia配置和檢修)
Hadoop illuminated - Open Source Hadoop Book(譯:Hadoop啟示——開源Hadoop書)
NoSQL Database(譯:NoSQL數據庫)
10 Best Practices for Apache Hive(譯:Apache Hive的10個最佳實踐)
Hadoop Operations at Scale
AWS BigData Blog(譯:AWS大數據博客)
Hadoop360
演示
Hadoop Summit Presentations - Slide decks from Hadoop Summit(譯:Hadoop的峰會展示)
Hadoop 24/7
An example Apache Hadoop Yarn upgrade(譯:Apache Hadoop Yarn更新實例)
Apache Hadoop In Theory And Practice(譯:Apache Hadoop的理論和實踐)
Hadoop Operations at LinkedIn(譯:Hadoop在LinkedIn的操作)
Hadoop Performance at LinkedIn(譯:Hadoop在LinkedIn的性能)
Docker based Hadoop provisioning(譯:基于Docker的Hadoop配置)
書籍
Hadoop: The Definitive Guide(譯:Hadoop權威指南)
Hadoop Operations(譯:Hadoop運營)
Apache Hadoop Yarn(譯:Apache Hadoop Yarn)
HBase: The Definitive Guide(譯:HBase權威指南)
Programming Pig(譯:Pig程序設計)
Programming Hive(譯:Hive程序設計)
Hadoop in Practice, Second Edition(譯:Hadoop實踐,第二版)
Hadoop in Action, Second Edition(譯:Hadoop實戰,第二版)
Hadoop&大數據事件
ApacheCon
Strata + Hadoop World
Hadoop Summit
以上為Hadoop學習資源集合的全部內容,更多精彩敬請期待。
若想獲取所有“史上最全Hadoop學習”資源,請關注云棲社區微信公眾號:yunqiinsight,并回復“史上最全Hadoop學習”。
*請認真填寫需求信息,我們會在24小時內與您取得聯系。