Page tree
Skip to end of metadata
Go to start of metadata

警告:Jupyter启动时会重置教程的笔记本。如果你想修改并保存它们,请复制一份。

初始化到Spark/Livy的连接

我们鼓励测试人员在他们自己的用户账户中运行这个笔记本。

代码:

from os import getenv
from scale.spark import connect

# 创建到Livy的连接
session = connect()

# 创建SparkSession的本地代理
spark = session.proxy('spark')

# 获取当前用户,防止用户覆盖其他用户的工作
USER = getenv('USER')

输出:

开环作业的关键参数

代码:

# 要运行的仿真数量
SAMPLE_SIZE = 30

# 仿真输入的位置(泛化空间)
STUDY_INPUT = '/hdfs/vtd/demodata/ScaleDriver-LaneChange/design_space_scenario.xml'

输出:

使用蒙特卡洛采样器创建样本

代码:

from scale.variation import load_xml, MonteCarlo

# 加载泛化空间
design_space = load_xml(STUDY_INPUT)

# 创建样本(Pandas数据帧)
sample = MonteCarlo(design_space, SAMPLE_SIZE).sample

sample

输出:


TgtTTC_StartLaneChangeTgtRateTgtSpeedSensorNearSensorFarSensorLeftSensorRightSensorBottomSensorTopSensorLatencyEgoSpeedEgoTimeGapEgoMinDistEgoBrakeAgilityEgoMaxAccelCmdEgoMaxDecelCmd
07.6191272.44925422.9243620.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
19.9212900.91319116.3181650.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
29.6677681.48725418.5151260.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
38.1123270.58399511.0689930.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
45.5014530.7520065.2789710.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
56.0674902.93914214.5970390.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
68.4959090.81769618.2075710.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
78.9081601.97187812.2487230.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
84.5485252.9765138.4785830.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
93.1646522.0742815.9588180.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
109.4494352.9365210.0580600.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
112.5831372.31095711.4981060.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
128.4514961.70163118.7909160.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
133.2633052.06588323.7304820.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
149.5546802.4971782.5123820.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
153.9816341.4726608.1486420.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
168.4994861.82840615.4750320.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
173.7447281.84390017.0791870.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
189.6918592.7581772.5030050.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
191.6923961.3018135.3721590.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
208.8227752.5061661.9126990.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
212.6704160.52755812.2947740.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
221.5520671.75425521.2451300.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
234.5685140.94404111.9718500.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
249.4812551.2552155.7014480.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
254.6448062.38120321.8429410.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
269.4352892.8513812.9214080.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
277.5293341.55248613.2172770.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
285.3447622.83459628.5520540.0150.010.010.05.05.02.040.01.010.00.76.0-9.81
297.1454031.7604450.5547130.0150.010.010.05.05.02.040.01.010.00.76.0-9.81

运行仿真

代码:

# 创建Spark数据帧
# 因为spark是一个代理,所以dataframe_input也是一个代理
dataframe_input = spark.createOpenLoopDataFrame(STUDY_INPUT, sample)

# 启动仿真
open_loop_results = dataframe_input \
    .setProject('scale://local/user/tutorials/03-Open-Loop') \
    .minibatch(10) \
    .simulate() \
    .copyFiles(copy_filter="failed-only") \
    .run(description=f'jupyter open loop notebook submitted by {USER}', save=True)

输出:

将数据帧保存为Parquet文件

代码:

(open_loop_results.write
     .mode('append')
     .parquet(f'/user/{USER}/results/vtd-tutorial3')
     .wait_for(only_wait=True))

输出:

waiting 0:00:01
waiting 0:00:02
waiting 0:00:05

化简、转换数据帧至Pandas、下载结果

代码:

# filter()只会过滤每次仿真的第一帧,否则数据帧将非常大。
# toPandas()会将Spark数据帧转换成Pandas数据帧。
# wait_for()会等待数值计算完成,然后从Livy转移到Jupyter(此处)。

pandas_df = open_loop_results.filter('frame==0').toPandas().wait_for()

输出:

waiting 0:00:02
value is ready and downloaded

交互式审查结果

要查看仿真时间,请尝试以下方法:

  1. 点击Type/Bar
  2. 对于Encoding/X,选择index
  3. 对于Encoding/Y,选择time_simulation
  4. 对于Encoding/Func.,选择Max

要查看TgtSpeed和安全因数之间的关系:

  1. 点击Type/Scatter
  2. 对于Encoding/X,选择TgtSpeed
  3. 对于Encoding/Y,选择kr_safety_factor
  4. 对于Encoding/Func.,选择Max


代码:

# 将Pandas数据帧可视化
from autovizwidget.widget.utils import display_dataframe
display_dataframe(pandas_df)

输出:

indexinput_uriinput_uri_shortparam_TgtTTC_StartLaneChangeparam_TgtRateparam_TgtSpeedparam_SensorNearparam_SensorFarparam_SensorLeftparam_SensorRight...kr_p3_xkr_p1_ykr_p2_zkr_p1_vykr_p3_vxresult_files_uritime_job_startdescriptiontime_job_durationjob_done
27/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange1.0886432.6239614.2217800.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin1392021-11-22 05:29:35.674584
7/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange2.5817890.8823870.8628010.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin1392021-11-22 05:29:35.674584
28/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange1.5457432.9996894.9156540.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin1392021-11-22 05:29:35.674584
11/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange7.0632130.85351116.9040080.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin1462021-11-22 05:29:35.674584
2/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange5.2566751.48539522.8311390.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin2742021-11-22 05:29:35.674584
0/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange8.7300441.94230311.8563100.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin2742021-11-22 05:29:35.674584
18/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange9.9934481.8938197.1755120.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin2742021-11-22 05:29:35.674584
26/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange4.6588301.2745078.5904880.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin2752021-11-22 05:29:35.674584
16/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange9.8226062.74885817.6076480.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin2752021-11-22 05:29:35.674584
29/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange1.0086812.61499229.7612680.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin2752021-11-22 05:29:35.674584
3/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange1.3718920.5317542.6204160.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin2842021-11-22 05:29:35.674584
9/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange4.6073012.63533527.8655260.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin3982021-11-22 05:29:35.674584
20/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange9.9028312.07444825.0531840.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin3982021-11-22 05:29:35.674584
5/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange8.2855872.45422422.3722340.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin3982021-11-22 05:29:35.674584
4/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange8.1609852.50572318.2434190.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin3982021-11-22 05:29:35.674584
24/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange8.6220251.06717216.1280100.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin4012021-11-22 05:29:35.674584
22/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange7.9692832.25361515.9746510.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin4012021-11-22 05:29:35.674584
13/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange3.5013531.89558110.8918330.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin4012021-11-22 05:29:35.674584
1/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange2.8656692.36601722.6290040.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin4092021-11-22 05:29:35.674584
19/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange1.8121440.98055016.6982220.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin4092021-11-22 05:29:35.674584
14/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange2.3413040.7634116.5406870.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin5372021-11-22 05:29:35.674584
10/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange7.5430832.60408927.0862170.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin5372021-11-22 05:29:35.674584
15/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange4.5295350.97380329.8791460.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin5372021-11-22 05:29:35.674584
17/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange1.4401412.88257323.0413710.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin5372021-11-22 05:29:35.674584
25/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange5.6787422.06222727.4511480.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin6482021-11-22 05:29:35.674584
23/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange8.7669081.6782424.2682030.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin6482021-11-22 05:29:35.674584
12/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange3.5807411.74085614.7849200.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin6482021-11-22 05:29:35.674584
21/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange6.3639350.62310824.5179480.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin6482021-11-22 05:29:35.674584
8/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange9.8429281.59967812.7360600.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin6482021-11-22 05:29:35.674584
6/hdfs/vtd/demodata/ScaleDriver-LaneChange/desi...ScaleDriver-LaneChange9.8186542.48332517.3921360.0150.010.010.0...NoneNoneNoneNoneNone
2021-11-22 05:18:47.820210jupyter open loop notebook submitted by admin6482021-11-22 05:29:35.674584

30 rows × 64 columns


Write a comment...