プログラム
1
Part1: 概要およびCaliperによるデータ⽣成 (60分)李在範( IMS Certified T&I Manager、ネットラーニング) Caliper概要 既存システムへのCaliper Sensor埋め込み
Part2: Caliperデータの取得と分析 (40分)常盤祐司(技術委員⻑、法政⼤学) AWSを⽤いた Caliper Recordの取得 AWSからPCへのデータダウンロード Tableau による分析
QA (20分)
Part2 Caliperデータの取得と分析
2
AWSを⽤いた Caliper Recordの取得
AWSからPCへのデータダウンロード
Tableau による分析
3
AWSを⽤いた Caliper Recordの取得
AWSからPCへのデータダウンロード
Tableau による分析
学習履歴データ分析概要
4
AmazonSQS
AmazonS3
AWSLambda
接続
SQS Pros & Cons
5
ProsS3 あるいは DB よりも⾼速書き込みができる。
Cons最⼤14⽇間しか保持されない。
Canvas設定
6
AWSのregionはCanvasと同様のus-east-1とした。
The name of the queue must begin with canvas-live-events.
取得する Caliper Profile 設定
7
AWS Simple Queue Service
8
SQS 設定
9
InstructureAccount Number
参考:https://community.canvaslms.com/docs/DOC-14163-how-do-i-create-an-sqs-queue-to-receive-live-events-data-from-canvas
SQSにおける蓄積
10
lambda
11
参考︓DevelopersIO、AWS LambdaがSQSをイベントソースとしてサポートしました︕https://dev.classmethod.jp/etc/aws-lambda-support-sqs-event-source/
IAM ロール
12
chaliceを⽤いてポリシーを設定する⽅法もある。
sqs-poll
13
lamda python
14
from __future__ import print_function# coding: utf-8import boto3 from datetime import datetime, timedelta, timezoneimport json
def lambda_handler(event, context):s3 = boto3.resource('s3')bucket = s3.Bucket('hosei-media-lambda')
JST = timezone(timedelta(hours=+9), 'JST')now = datetime.now(JST).strftime("%Y%m%d-%H%M%S-%f")
for i,record in enumerate(event['Records']):file_contents = record["body"]fn = 'sqs-poll_' + str(now) + '_' + str(i)ret = bucket.put_object(ACL='private',Body=file_contents,Key = fn,ContentType='text/plain')
return
s3コンソール
15
レコード概要
16
s3レコード表⽰
17
18
AWSを⽤いた Caliper Recordの取得
AWSからPCへのデータダウンロード
Tableau による分析
s3からのダウンロード
19
aws s3 cp --region us-east-1 s3://hosei-media-lambda/ . --recursive
AWS CLI によるCurrent Directoryへのレコードダウンロード
20
{"sensor": "http://hosei.instructure.com/","sendTime": "2019-09-07T07:09:40.074Z","dataVersion": "http://purl.imsglobal.org/ctx/caliper/v1p1","data": [{
"@context": "http://purl.imsglobal.org/ctx/caliper/v1p1","id": "urn:uuid:ae1587d5-493e-4c8c-9435-3c8c22cb024e","type": "SessionEvent","actor": {
"id": "urn:instructure:canvas:user:125310000000000006","type": "Person","extensions": {
"com.instructure.canvas": {"user_login": "[email protected]","root_account_id": "125310000000000001","root_account_lti_guid": "tkMMgk9o7kftOu2oN50lbvwXxSPJM9yjcYsRIE0p:canvas-lms","root_account_uuid": "tkMMgk9o7kftOu2oN50lbvwXxSPJM9yjcYsRIE0p","entity_id": "125310000000000006"
}}
},"action": "LoggedIn","object": {
"id": "http://hosei.instructure.com/","type": "SoftwareApplication"
},"eventTime": "2019-09-07T07:09:39.764Z","edApp": {
"id": "http://hosei.instructure.com/","type": "SoftwareApplication"
},"session": {
"id": "urn:instructure:canvas:session:f66e564864e641476657696e0734915f","type": "Session"
},"extensions": {
"com.instructure.canvas": {"hostname": "hosei.instructure.com","request_id": "68ce4f5d-615e-408f-bbfb-46ec08008eef","user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36","client_ip": "133.25.246.231","request_url": "https://hosei.instructure.com/login/canvas","version": "1.0.0"
}}
}]}
Caliper Data
配列
JSON全データ読込⽤Python事例
21
全データ読込⽤Pythonプログラム事例# -*- coding: utf-8 -*-# python r_20.py caliper.json# 2018-8-16 Y.Tokiwa
import jsonimport sys
args = sys.argv
def keyGet(k,v): if isinstance(v,dict):
print("Dict-> ",k)for k,v1 in v.items():
keyGet(k,v1)return
# "data"keyに対して配列(=list)を想定しているための処理elif isinstance(v,list):
print("List-> ",k)
# listの形式"data"からjsonをv1に取り出すfor num,v1 in enumerate(v):
keyGet(k,v1)return
# ここでキーとバリューを書き出す。else:
print(k,":",v)return
if __name__ == "__main__":with open(args[1], encoding='utf-8') as f:
record = json.load(f)keyGet("json",record)
{"sensor": "multiverse-course-1003032","sendTime": "2018-06-25T05:00:01.6857809Z","data": [
{"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context","@type": "http://purl.imsglobal.org/caliper/v1/AssessmentEvent","actor": {
"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context","@id": "https://nlp.netlearning.co.jp/user/14383282","@type": "http://purl.imsglobal.org/caliper/v1/lis/Person","name": "LTI00000031","description": null,"extensions": {},"dateCreated": null,"dateModified": "2018-06-18T09:26:11Z"
},"action": "http://purl.imsglobal.org/vocab/caliper/v1/action#Started","object": {
"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",…
$ winpty python r_20.py nobom.jsonDict-> jsonsensor : multiverse-course-1003032sendTime : 2018-06-25T05:00:01.6857809ZList-> dataDict-> data@context : http://purl.imsglobal.org/ctx/caliper/v1/Context@type : http://purl.imsglobal.org/caliper/v1/AssessmentEventDict-> actor@context : http://purl.imsglobal.org/ctx/caliper/v1/Context@id : https://nlp.netlearning.co.jp/user/14383282@type : http://purl.imsglobal.org/caliper/v1/lis/Personname : LTI00000031description : NoneDict-> extensionsdateCreated : NonedateModified : 2018-06-18T09:26:11Zaction : http://purl.imsglobal.org/vocab/caliper/v1/action#StartedDict-> object@context : http://purl.imsglobal.org/ctx/caliper/v1/Context…
Caliper v1.0 JSON
Output
Tableau⽤データ⽣成
22
Tableauデータ出⼒⽤Pythonプログラム事例user_loginがあるデータのみ抽出
# -*- coding: utf-8 -*-# 2018-8-16 Y.Tokiwa
import jsonimport sys
import globimport os
from datetime import datetimeimport pytzimport sys
args = sys.argv
def keyGet(dict):for k,v in dict.items():
keyGet1(k,v)return
def keyGet1(k,v):global ind_user_login, user_login, eventTime, action, asset_type, name
if isinstance(v,str):if k == "user_login":
user_login = vind_user_login = 1
if ind_user_login == 1:if k == "eventTime": eventTime = v if k == "action": action = v if k == "asset_type" :asset_type = v if k == "name": name = v
if isinstance(v,dict):# print("Dict-> ",k)return keyGet(v)
def iso_to_jstdt(iso_str):dt = Nonetry:
dt = datetime.strptime(iso_str, '%Y-%m-%dT%H:%M:%S.%fZ')dt = pytz.utc.localize(dt).astimezone(pytz.timezone("Asia/Tokyo"))
except ValueError:try:
dt = datetime.strptime(iso_str, '%Y-%m-%dT%H:%M:%S.%f%z')dt = dt.astimezone(pytz.timezone("Asia/Tokyo"))
except ValueError:pass
return dt
if __name__ == "__main__":
s3_directory = "./"+args[1]+"/*"s3_files = glob.glob(s3_directory)
for s3file in s3_files:with open(s3file, encoding='utf-8') as f:
ind_user_login = 0user_login = ""eventTime = ""action = ""asset_type = ""name = ""data = json.load(f)keyGet(data["data"][0])if ind_user_login == 1:
print(iso_to_jstdt(eventTime), user_login, action, asset_type, name,sep=",")続く
Tableau⽤データ事例
23
2019-08-06 13:19:29.156000+09:00,[email protected],Modified,,2019-08-06 13:19:29.470000+09:00,[email protected],Modified,,Report32019-08-06 13:20:38.110000+09:00,[email protected],NavigatedTo,course,home2019-08-06 13:20:42.358000+09:00,[email protected],NavigatedTo,course,announcements2019-08-06 13:25:57.934000+09:00,[email protected],NavigatedTo,discussion_topic,2019-08-06 13:25:58.018000+09:00,[email protected],Created,,Report5 返却および期末テストを除いた成績のお知らせ2019-08-06 13:25:58.285000+09:00,[email protected],NavigatedTo,discussion_topic,2019-08-06 13:26:08.521000+09:00,[email protected],NavigatedTo,course,roster2019-08-06 13:27:28.553000+09:00,[email protected],NavigatedTo,course,outcomes2019-08-06 13:27:34.110000+09:00,[email protected],LoggedOut,,2019-08-06 13:52:51.928000+09:00,15X0001,NavigatedTo,course,home2019-08-06 15:07:32.991000+09:00,17X0001,NavigatedTo,course,home2019-08-06 15:07:41.194000+09:00,17X0001,NavigatedTo,course,home2019-08-06 17:06:05.952000+09:00,17X0001,NavigatedTo,course,home2019-08-06 17:06:16.572000+09:00,17X0001,NavigatedTo,course,home2019-08-06 18:04:15.548000+09:00,16X0001,NavigatedTo,course,home2019-08-06 18:04:24.205000+09:00,16X0001,NavigatedTo,course,home2019-08-06 19:41:41.927000+09:00,17X0002,NavigatedTo,course,home2019-08-06 19:41:52.515000+09:00,17X0002,NavigatedTo,course,home2019-08-06 20:06:26.529000+09:00,17X0002,LoggedIn,,2019-08-06 20:06:26.848000+09:00,17X0002,NavigatedTo,course,home2019-08-06 20:06:27.556000+09:00,17X0002,LoggedIn,,2019-08-06 20:06:31.622000+09:00,17X0002,NavigatedTo,course,home2019-08-06 20:06:38.494000+09:00,17X0002,NavigatedTo,course,grades2019-08-07 13:49:01.311000+09:00,[email protected],LoggedIn,,2019-08-07 13:49:06.044000+09:00,[email protected],NavigatedTo,course,home2019-08-07 13:49:22.107000+09:00,[email protected],NavigatedTo,course,roster
次のスライドにこのデータを⽣成した元のCaliper
データを⽰す
24
{"sensor": "http://hosei.instructure.com/","sendTime": "2019-08-06T04:25:59.834Z","dataVersion": "http://purl.imsglobal.org/ctx/caliper/v1p1","data": [{
"@context": "http://purl.imsglobal.org/ctx/caliper/v1p1","id": "urn:uuid:3d5cf58b-1ef1-46df-bd83-13af7483be65","type": "ThreadEvent","actor": {
"id": "urn:instructure:canvas:user:125310000000000006","type": "Person","extensions": {
"com.instructure.canvas": {"user_login": "[email protected]","root_account_id": "125310000000000001","root_account_lti_guid": "tkMMgk9o7kftOu2 …YsRIE0p:canvas-lms","root_account_uuid": "tkMMgk9o7kftOu2 …. YsRIE0p","entity_id": "125310000000000006"
}}
},"action": "Created","object": {
"id": "urn:instructure:canvas:discussion:125310000000000099","type": "Thread","name": "Report5 返却および期末テストを除いた成績のお知らせ","extensions": {
"com.instructure.canvas": {"is_announcement": true,"entity_id": "125310000000000099"
}}
},"eventTime": "2019-08-06T04:25:58.018Z","edApp": {
"id": "http://hosei.instructure.com/","type": "SoftwareApplication"
},
"group": {"id": "urn:instructure:canvas:course:46","type": "CourseOffering","extensions": {
"com.instructure.canvas": {"context_type": "Course","entity_id": "46"
}}
},"membership": {
"id": "urn:instructure:canvas:course:46:Instructor:125310000000000006","type": "Membership","member": {
"id": "urn:instructure:canvas:user:125310000000000006","type": "Person"
},"organization": {
"id": "urn:instructure:canvas:course:46","type": "CourseOffering"
},"roles": ["Instructor"]
},"session": {
"id": "urn:instructure:canvas:session:93c6587b79128ee5cd6dd0ef9accb136","type": "Session"
},"extensions": {
"com.instructure.canvas": {"hostname": "hosei.instructure.com","request_id": "3aee1472-6dc1-4b56-aedb-85e655e3bea5","user_agent": "Mozilla/5.0 …. Chrome/75.0.3770.142 Safari/537.36","client_ip": "133.25.246.231","request_url": "https://hosei.instructure.com/api/v1/courses/46/discussion_topics","version": "1.0.0"
}}
}]}
Tableauデータを⽣成したCaliper v1.1 Data 事例
続く
25
AWSを⽤いた Caliper Recordの取得
AWSからPCへのデータダウンロード
Tableau による分析
Why Tableau
26
Amazon Redshift
アカデミックプログラム
27
LMSで記録された学習履歴サマリー
28
Tableauデータ読み込み
29
Tableauイベントの表⽰
30
Tableau分析結果まとめ
31
データソース置換
32
33
Top Related