sbt, the interactive build tool
http://scala-sbt.org
sbt is a build tool for Scala, Java, and more.
For general documentation, see http://www.scala-sbt.org/.
sbt is a build tool for Scala, Java, and more.
For general documentation, see http://www.scala-sbt.org/.
sbt 1.0.x
This is the 1.0.x series of sbt. The source code of sbt is split across several Github repositories, including this one.- sbt/io hosts
sbt.iomodule. - sbt/util hosts a collection of internally used modules.
- sbt/librarymanagement hosts
sbt.librarymanagementmodule that wraps Ivy. - sbt/zinc hosts Zinc, an incremental compiler for Scala.
- sbt/sbt, this repository hosts modules that implements the build tool.
Other links
- Setup: Describes getting started with the latest binary release.
- FAQ: Explains how to get help and more.
- sbt/sbt-zero-seven: hosts sbt 0.7.7 and earlier versions
(https://briteming.blogspot.com/2014/05/sbt.html)
-------
SBT&Spark使用
Spark项目打包和发布
###SBT 常用命令
actions – 显示对当前工程可用的命令
update – 下载依赖
compile – 编译代码
test – 运行测试代码
package – 创建一个可发布的jar包
publish-local – 把构建出来的jar包安装到本地的ivy缓存
publish – 把jar包发布到远程仓库（如果配置了的话)
sbt-assembly
使用sbt-assembly打jar主要有两个版本0.11.2和0.13.0
对于0.11.2版
在project目录加 assembly.sbt 写入:
$ cat project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
然后在项目根目录加入 build.sbt ，因为spark包含了scala库所以不需要再次包含。
import AssemblyKeys._
name := "Simple Project"
version := "1.0"
organization := "com.databricks"
scalaVersion := "2.10.4"
// 加入第三方依赖
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-client" % "2.3.0" % "provided",
"org.apache.spark" %% "spark-core" % "1.3.0" % "provided"
// Third-party libraries
"net.sf.jopt-simple" % "jopt-simple" % "4.3",
"joda-time" % "joda-time" % "2.0"
)
// This statement includes the assembly plug-in capabilities
assemblySettings
// Configure JAR used with the assembly plug-in
jarName in assembly := "my-project-assembly.jar"
// A special option to exclude Scala itself form our assembly JAR, since Spark
// already bundles Scala.
assemblyOption in assembly :=
(assemblyOption in assembly).value.copy(includeScala = false)
对于0.13.0版
在project目录加 assembly.sbt 写入:
$ cat project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
然后在项目根目录加入 build.sbt ，因为spark包含了scala库所以不需要再次包含。
name := "hello"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-client" % "2.3.0" % "provided",
"org.apache.spark" %% "spark-core" % "1.3.0" % "provided"
)
// Configure JAR used with the assembly plug-in
assemblyJarName in assembly := "wordcount-assembly.jar"
// A special option to exclude Scala itself form our assembly JAR, since Spark
// already bundles Scala.
assemblyOption in assembly :=
(assemblyOption in assembly).value.copy(includeScala = false)
然后使用 sbt assembly就可以打jar包了。
> assembly
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[info] Assembly up to date: /usr/home/service/projects/hello/target/scala-2.11/wordcount-assembly.jar
[success] Total time: 0 s, completed 2015-4-2 11:27:43
使用eclipse sbt 插件
在相关的sbt用户目录加入全局插件配置:
// ~\.sbt\0.13\plugins global plugin dir
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "3.0.0")
然后就可以使用 sbt eclipse构建初始化eclipse项目
>> sbt eclipse
[info] About to create Eclipse project files for your project(s).
[info] Successfully created Eclipse project files for project(s):
[info] app
依赖导出
在build.sbt中设置:
retrieveManaged := true
SBT打包Java应用
使用sbt管理java项目，build.sbt配置如下
// Project name (artifact name in Maven)
name := "$Project_name"
// orgnization name (e.g., the package name of the project)
organization := "$Organization"
version := "1.0-SNAPSHOT"
// project description
description := "Treasure Data Project"
// Enables publishing to maven repo
publishMavenStyle := true
// Do not append Scala versions to the generated artifacts
crossPaths := false
// This forbids including Scala related libraries into the dependency
autoScalaLibrary := false
// library dependencies. (orginization name) % (project name) % (version)
libraryDependencies ++= Seq(
"org.apache.commons" % "commons-math3" % "3.1.1",
"org.fluentd" % "fluent-logger" % "0.2.10",
"org.mockito" % "mockito-core" % "1.9.5" % "test" // Test-only dependency
)
Maven打包Java应用
POM配置
<project>
<modelVersion>4.0.0</modelVersion>
<!-- Information about your project -->
<groupId>com.databricks</groupId>
<artifactId>example-build</artifactId>
<name>Simple Project</name>
<packaging>jar</packaging>
<version>1.0</version>
<dependencies>
<!-- Spark dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.2.0</version>
<scope>provided</scope>
</dependency>
<!-- Third-party library -->
<dependency>
<groupId>net.sf.jopt-simple</groupId>
<artifactId>jopt-simple</artifactId>
<version>4.3</version>
</dependency>
<!-- Third-party library -->
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<!-- Maven shade plug-in that creates uber JARs -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
Packaging Your Code and Dependencies | 125
</executions>
</plugin>
</plugins>
</build>
</project>
打包命令
$ mvn package
# In the target directory, we'll see an uber JAR and the original package JAR
$ ls target/
example-build-1.0.jar
original-example-build-1.0.jar
# Listing the uber JAR will reveal classes from dependency libraries
$ jar tf target/example-build-1.0.jar
...
joptsimple/HelpFormatter.class
...
org/joda/time/tz/UTCProvider.class
...
# An uber JAR can be passed directly to spark-submit
$ /path/to/spark/bin/spark-submit --master local ... target/example-build-1.0.jar
部署提交
spark的app通过spark的spark-sumbit提交任务可以提交python，java，scala写的程序。
提交 Python 应用
bin/spark-submit my_script.py
Using spark-submit with various options
# Submitting a Java application to Standalone cluster mode
$ ./bin/spark-submit \
--master spark://hostname:7077 \
--deploy-mode cluster \
--class com.databricks.examples.SparkExample \
--name "Example Program" \
--jars dep1.jar,dep2.jar,dep3.jar \
--total-executor-cores 300 \
--executor-memory 10g \
myApp.jar "options" "to your application" "go here"
# Submitting a Python application in YARN client mode
$ export HADOP_CONF_DIR=/opt/hadoop/conf
$ ./bin/spark-submit \
--master yarn \
--py-files somelib-1.2.egg,otherlib-4.4.zip,other-file.py \
--deploy-mode client \
--name "Example Program" \
--queue exampleQueue \
--num-executors 40 \
--executor-memory 10g \
my_script.py "options" "to your application" "go here"
