|
Hartynspren
Angemeldet seit:
02.01.2023
Beiträge: 178
|
In this notebook we show how to obtain crypto-currencies data from several data sources and make some basic time series visualizations. We assume the described data acquisition workflow is useful for doing more detailed (exploratory) analysis.
There are multiple crypto-currencies data sources, but a small proportion of them give a convenient way of extracting crypto-currencies data automatically. I found the easiest to work with to be ***** [YF1]. Another easy to work with Bitcoin-only data source is ***** , [DBO1].
(I also looked into using ***** )
Remark: The code below is made with certain ad-hoc inductive reasoning that brought meaningful results. This means the code has to be changed if the underlying data organization in [YF1, DBO1] is changed.
Yahoo! Finance
Getting cryptocurrencies symbols and summaries
In this section we get all crypto-currencies symbols and related metadata.
Get the data of all crypto-currencies in [YF1]:
In[]:=
AbsoluteTiming[lsData=Import["https://finance.yahoo.com/cryptocurrencies","Data"];]
Out[]=
{6.18067,Null}
Locate the data:
In[]:=
pos=First@Position[lsData,{"Symbol","Name","Price(Intraday)","Change","%Change",___}];dsCryptoCurrenciesColumnNames=lsData〚Sequence@@pos〛Length[dsCryptoCurrenciesColumnNames]
Out[]=
{Symbol,Name,Price(Intraday),Change,%Change,MarketCap,VolumeinCurrency(Since0:00UTC),VolumeinCurrency(24Hr),TotalVolumeAllCurrencies(24Hr),CirculatingSupply,52WeekRange,1DayChart}
Out[]=
12
Get the data:
In[]:=
dsCryptoCurrencies=lsData〚Sequence@@Append[Most[pos],2]〛;Dimensions[dsCryptoCurrencies]
Out[]=
{25,10}
Make a dataset:
In[]:=
dsCryptoCurrencies=Dataset[dsCryptoCurrencies][All,AssociationThread[dsCryptoCurrenciesColumnNames〚1;;-3〛,#]&]
Out[]=
Symbol
Name
Price(Intraday)
Change
%Change
MarketCap
VolumeinCurrency(Since0:00UTC)
VolumeinCurrency(24Hr)
TotalVolumeAllCurrencies(24Hr)
CirculatingSupply
BTC-USD
BitcoinUSD
32,624.94
-2,453.94
-7.00%
611.392B
50.638B
50.638B
50.638B
18.74M
ETH-USD
EthereumUSD
1,938.09
-233.97
-10.77%
225.563B
32.17B
32.17B
32.17B
116.385M
USDT-USD
TetherUSD
1.0007
-0.0006
-0.06%
62.678B
86.511B
86.511B
86.511B
62.631B
BNB-USD
BinanceCoinUSD
298.43
-33.52
-10.10%
45.79B
2.797B
2.797B
2.797B
153.433M
ADA-USD
CardanoUSD
1.2597
-0.1703
-11.91%
40.241B
3.67B
3.67B
3.67B
31.946B
XRP-USD
XRPUSD
0.664478
-0.079616
-10.70%
30.703B
3.84B
3.84B
3.84B
46.206B
DOGE-USD
DogecoinUSD
0.208697
-0.067533
-24.45%
27.155B
4.692B
4.692B
4.692B
130.116B
USDC-USD
USDCoinUSD
1.0007
-0.0007
-0.07%
24.96B
4.034B
4.034B
4.034B
24.942B
DOT1-USD
PolkadotUSD
16.93
-3.39
-16.67%
16.152B
2.244B
2.244B
2.244B
954.06M
HEX-USD
HEXUSD
0.083123
-0.007879
-8.66%
14.414B
38.896M
38.896M
38.896M
173.411B
UNI3-USD
UniswapUSD
16.96
-3.1
-15.45%
9.755B
534.06M
534.06M
534.06M
575.171M
BCH-USD
BitcoinCashUSD
481.24
-61.1
-11.27%
9.033B
2.054B
2.054B
2.054B
18.77M
LTC-USD
LitecoinUSD
133.55
-18.09
-11.93%
8.915B
2.797B
2.797B
2.797B
66.752M
LINK-USD
ChainlinkUSD
18.5
-2.44
-11.67%
8.028B
1.891B
1.891B
1.891B
434.01M
SOL1-USD
SolanaUSD
28.31
-6.39
-18.41%
7.719B
705.256M
705.256M
705.256M
272.637M
MATIC-USD
MaticNetworkUSD
1.2177
-0.167
-12.06%
7.663B
1.438B
1.438B
1.438B
6.293B
THETA-USD
THETAUSD
7.2805
-1.3067
-15.22%
7.28B
368.597M
368.597M
368.597M
1B
XLM-USD
StellarUSD
0.25862
-0.034421
-11.75%
5.992B
652.713M
652.713M
652.713M
23.17B
VET-USD
VeChainUSD
0.08124
-0.014682
-15.31%
5.225B
710.449M
710.449M
710.449M
64.316B
ICP1-USD
InternetComputerUSD
39.4
-8.71
-18.11%
5.082B
238.555M
238.555M
238.555M
128.986M
rows1–20of25
Get all time series
In this section we get all the crypto-currencies time series from [YF1].
In[]:=
AbsoluteTiming[ccNow=Round@AbsoluteTime[Date[]]-AbsoluteTime[{1970,1,1,0,0,0}];aCryptoCurrenciesDataRaw=Association@Map[#->ResourceFunction["ImportCSVToDataset"]["https://query1.finance.yahoo.com/v7/finance/download/"<>#<>"?period1=1410825600&period2="<>ToString[ccNow]<>"&interval=1d&events=history&includeAdjustedClose=true"]&,Normal[dsCryptoCurrencies[All,"Symbol"]]];]
Out[]=
{5.98745,Null}
Remark: Note that in the code above we specified the upper limit of the time span to be the current date. (And shifted it with respect to the epoch start 1970-01-01 used by [YF1].)
Check we good the data with dimensions retrieval:
In[]:=
Dimensions/@aCryptoCurrenciesDataRaw
Out[]=
BTC-USD{2468,7},ETH-USD{2144,7},USDT-USD{2307,7},BNB-USD{1426,7},ADA-USD{1358,7},DOGE-USD{2468,7},XRP-USD{2468,7},USDC-USD{986,7},DOT1-USD{304,7},HEX-USD{551,7},UNI3-USD{81,7},BCH-USD{1428,7},LTC-USD{2468,7},SOL1-USD{436,7},LINK-USD{1369,7},THETA-USD{1250,7},MATIC-USD{784,7},XLM-USD{2468,7},ICP1-USD{32,7},VET-USD{1052,7},ETC-USD{1792,7},FIL-USD{1285,7},TRX-USD{1376,7},XMR-USD{2468,7},EOS-USD{1450,7}
Check we good the data with random sample:
In[]:=
RandomSample[#,6]&/@KeyTake[aCryptoCurrenciesDataRaw,RandomChoice[Keys@aCryptoCurrenciesDataRaw]]
Out[]=
Here we add the crypto-currencies symbols and convert date strings into date objects.
In[]:=
AbsoluteTiming[aCryptoCurrenciesData=Association@KeyValueMap[Function[{k,v},k->v[All,Join[<|"Symbol"->k,"DateObject"->DateObject[#Date]|>,#]&]],aCryptoCurrenciesDataRaw];]
Out[]=
{8.27865,Null}
Summary
In this section we compute the summary over all datasets:
In[]:=
ResourceFunction["RecordsSummary"][Join@@Values[aCryptoCurrenciesData],"MaxTallies"->30]
Out[]=
1Symbol
BTC-USD
2468
DOGE-USD
2468
LTC-USD
2468
XLM-USD
2468
XMR-USD
2468
XRP-USD
2468
USDT-USD
2307
ETH-USD
2144
ETC-USD
1792
EOS-USD
1450
BCH-USD
1428
BNB-USD
1426
TRX-USD
1376
LINK-USD
1369
ADA-USD
1358
FIL-USD
1285
THETA-USD
1250
VET-USD
1052
USDC-USD
986
MATIC-USD
784
HEX-USD
551
SOL1-USD
436
DOT1-USD
304
UNI3-USD
81
ICP1-USD
32
,
2DateObject
Min
Wed17Sep201400:00:00GMT-4
1stQu
Thu28Sep201700:00:00GMT-4
Mean
Sat24Nov201806:14:19GMT-4
Median
Tue5Mar201900:00:00GMT-4
3rdQu
Sat23May202000:00:00GMT-4
Max
Sat19Jun202100:00:00GMT-4
,
3Date
2021-05-19
25
2021-05-20
25
2021-05-21
25
2021-05-22
25
2021-05-23
25
2021-05-24
25
2021-05-25
25
2021-05-26
25
2021-05-27
25
2021-05-28
25
2021-05-29
25
2021-05-30
25
2021-05-31
25
2021-06-01
25
2021-06-02
25
2021-06-03
25
2021-06-04
25
2021-06-05
25
2021-06-06
25
2021-06-07
25
2021-06-08
25
2021-06-09
25
2021-06-10
25
2021-06-11
25
2021-06-12
25
2021-06-13
25
2021-06-14
25
2021-06-15
25
2021-06-16
25
(Other)
35494
,
4Open
null
723
1.
563
1.00001
53
0.999999
36
0.000228
23
0.000223
19
0.000208
17
0.00013
16
0.000231
15
0.000215
14
0.000224
14
1.00002
14
0.000142
13
0.000219
13
0.000221
13
0.000222
13
0.000226
13
0.000227
13
0.00023
12
0.000141
11
0.000144
11
0.000216
11
0.000225
11
0.000126
10
0.000128
10
0.000129
10
0.000143
10
0.000211
10
0.000212
10
(Other)
34518
,
5High
null
723
1.
564
1.0002
48
1.00001
47
0.999999
34
0.000228
23
0.000231
15
0.000235
15
0.000224
14
0.00023
13
0.000234
13
1.00002
13
0.000132
12
0.000134
12
0.000147
12
0.000233
12
0.00024
12
0.000133
11
0.000145
11
0.00021
11
0.000212
11
0.000217
11
0.000218
11
0.00022
11
0.000226
11
0.000229
11
0.000239
11
0.000131
10
0.000215
10
(Other)
34507
,
6Low
null
723
1.
555
1.00001
53
0.9998
48
0.999999
33
0.000224
21
0.000223
20
0.000219
17
0.000128
16
0.000218
16
0.000222
16
0.00022
15
0.000221
15
0.999801
15
0.000139
14
0.00014
14
0.000202
14
0.000225
14
0.000127
13
0.000232
13
0.000142
12
0.000215
12
0.000216
12
0.000227
12
0.000233
12
0.000235
12
0.999998
12
0.000138
11
0.000203
11
(Other)
34468
,
7Close
null
723
1.
563
1.00001
53
0.999999
35
0.000208
17
0.000226
17
0.000228
17
0.000224
16
0.00013
15
0.000229
15
0.000209
14
0.000221
14
0.000223
14
0.000225
14
0.00023
14
1.00002
14
0.000141
13
0.00022
13
0.000129
12
0.000214
12
0.000219
12
0.999998
12
0.000142
11
0.000211
11
0.000238
11
0.000131
10
0.000143
10
0.000144
10
0.000217
10
(Other)
34517
,
8AdjClose
null
723
1.
563
1.00001
53
0.999999
35
0.000208
17
0.000226
17
0.000228
17
0.000224
16
0.00013
15
0.000229
15
0.000209
14
0.000221
14
0.000223
14
0.000225
14
0.00023
14
1.00002
14
0.000141
13
0.00022
13
0.000129
12
0.000214
12
0.000219
12
0.999998
12
0.000142
11
0.000211
11
0.000238
11
0.000131
10
0.000143
10
0.000144
10
0.000217
10
(Other)
34517
,
9Volume
null
723
0
21
31
3
50
3
51
3
15499400
3
3
2
4
2
5
2
6
2
21
2
22
2
30
2
37
2
49
2
69
2
92
2
323
2
1207
2
4101
2
4634
2
5382
2
6058
2
9284
2
15430
2
21904
2
22508
2
23255
2
24533
2
(Other)
35417
Plots
Here we plot the “Low” and “High” price time series for each crypto-currency for the last 120 days:
In[]:=
nDays=120;Map[Block[{dsTemp=#[Select[AbsoluteTime[#DateObject]>AbsoluteTime[DatePlus[Now,-Quantity[nDays,"Days"]]]&]]},DateListPlot[{Normal[dsTemp[All,{"DateObject","Low"}][Values]],Normal[dsTemp[All,{"DateObject","High"}][Values]]},PlotLegends{"Low","High"},AspectRatio1/4,PlotRangeAll]]&,aCryptoCurrenciesData]
Out[]=
Here we plot the volume time series for each crypto-currency for the last 120 days:
In[]:=
nDays=120;Map[Block[{dsTemp=#[Select[AbsoluteTime[#DateObject]>AbsoluteTime[DatePlus[Now,-Quantity[nDays,"Days"]]]&]]},DateListPlot[{Normal[dsTemp[All,{"DateObject","Volume"}][Values]]},PlotLabel"Volume",AspectRatio1/4,PlotRangeAll]]&,aCryptoCurrenciesData]
Out[]=
data.bitcoinity.org
In this section we ingest crypto-currency data from data.bitcoinity.org, [DBO1].
Metadata
In this sub-section we assign different metadata elements used in data.bitcoinity.org.
The currencies and exchanges we obtained by examining the output of:
In[]:=
Import["https://data.bitcoinity.org/markets/price/30d/USD?t=l", "Plaintext"]
Assignments
In[]:=
lsCurrencies={"all","AED","ARS","AUD","BRL","CAD","CHF","CLP","CNY","COP","CZK","DKK","EUR","GBP","HKD","HRK","HUF","IDR","ILS","INR","IRR","JPY","KES","KRW","MXN","MYR","NOK","NZD","PHP","PKR","PLN","RON","RUB","RUR","SAR","SEK","SGD","THB","TRY","UAH","USD","VEF","XAU","ZAR"};
In[]:=
lsExchanges={"all","bit-x","bit2c","bitbay","bitcoin.co.id","bitcoincentral","bitcoinde","bitcoinsnorway","bitcurex","bitfinex","bitflyer","bithumb","bitmarketpl","bitmex","bitquick","bitso","bitstamp","btcchina","btce","btcmarkets","campbx","cex.io","clevercoin","coinbase","coinfloor","exmo","gemini","hitbtc","huobi","itbit","korbit","kraken","lakebtc","localbitcoins","mercadobitcoin","okcoin","paymium","quadrigacx","therocktrading","vaultoro","wallofcoins"};
In[]:=
lsTimeSpans={"10m","1h","6h","24h","3d","30d","6m","2y","5y","all"};
In[]:=
lsTimeUnit={"second","minute","hour","day","week","month"};
In[]:=
aDataTypeDescriptions=Association@{"price""Prince","volume""TradingVolume","rank""Rank","bidask_sum""Bid/AskSum","spread""Bid/AskSpread","tradespm""TradesPerMinute"};lsDataTypes=Keys[aDataTypeDescriptions];
Getting BTC data
Here we make a template string that for CSV data retrieval from data.bitcoinity.org:
In[]:=
stDBOURL=StringTemplate["https://data.bitcoinity.org/export_data.csv?currency=`currency`&data_type=`dataType`&exchange=`exchange`&r=`timeUnit`&t=l×pan=`timeSpan`"]
Out[]=
TemplateObject[{https://data.bitcoinity.org/export_data.csv?currency=,TemplateSlot[currency],&data_type=,TemplateSlot[dataType],&exchange=,TemplateSlot[exchange],&r=,TemplateSlot[timeUnit],&t=l×pan=,TemplateSlot[timeSpan]},CombinerFunctionStringJoin,InsertionFunctionTextString]
Here is an association with default values for the string template above:
In[]:=
aDBODefaultParameters=<|"currency"->"USD","dataType"->"price","exchange"->"all","timeUnit"->"day","timeSpan"->"all"|>;
Remark: The metadata assigned above is used to form valid queries for the query string template.
Remark: Not all combinations of parameters are “fully respected” by data.bitcoinity.org. For example, if a data request is with time granularity that is too fine over a large time span, then the returned data is with coarser granularity.
Price for a particular currency and exchange pair
Here we retrieve data by overwriting the parameters for currency, time unit, time span, and exchange:
In[]:=
dsBTCPriceData=ResourceFunction["ImportCSVToDataset"][stDBOURL[Join[aDBODefaultParameters,<|"currency"->"EUR","timeUnit"->"hour","timeSpan"->"30d","exchange"->"coinbase"|>]]]
Out[]=
Here is a summary:
In[]:=
ResourceFunction["RecordsSummary"][dsBTCPriceData]
Out[]=
1Time
2021-06-1216:00:00UTC
1
2021-06-1217:00:00UTC
1
2021-06-1218:00:00UTC
1
2021-06-1219:00:00UTC
1
2021-06-1220:00:00UTC
1
2021-06-1221:00:00UTC
1
(Other)
161
,
2avg
Min
28993.5
1stQu
30340.9
Mean
31737.
Median
32143.5
3rdQu
32937.1
Max
33846.8
,
3max
Min
29188.9
1stQu
30658.1
Mean
31938.2
Median
32380.2
3rdQu
33090.
Max
34100.
,
4min
Min
28822.9
1stQu
30118.1
Mean
31539.5
Median
31955.
3rdQu
32733.1
Max
33430.
Volume data
Here we retrieve data by overwriting the parameters for data type, time unit, time span, and exchange:
In[]:=
dsBTCVolumeData=ResourceFunction["ImportCSVToDataset"][stDBOURL[Join[aDBODefaultParameters,<|"dataType"->Volume,"timeUnit"->"day","timeSpan"->"30d","exchange"->"all"|>]]]
Out[]=
Here is a summary:
In[]:=
ResourceFunction["RecordsSummary"][dsBTCVolumeData]
Out[]=
1Time
2021-05-2000:00:00UTC
1
2021-05-2100:00:00UTC
1
2021-05-2200:00:00UTC
1
2021-05-2300:00:00UTC
1
2021-05-2400:00:00UTC
1
2021-05-2500:00:00UTC
1
(Other)
24
,
2bit-x
25
0.0001
1
0.0006
1
0.0023
1
0.005
1
0.0055
1
,
3bitfinex
Min
3538.42
1stQu
6016.74
Median
7766.35
Mean
10306.9
3rdQu
11411.1
Max
26969.6
,
4bitstamp
Min
2526.73
1stQu
4058.94
Median
4366.27
Mean
5692.08
3rdQu
6167.02
Max
13668.5
,
5cex.io
Min
107.51
1stQu
199.933
Median
224.479
Mean
322.398
3rdQu
333.374
Max
922.877
,
6coinbase
Min
8184.5
1stQu
15749.
Median
19459.
Mean
24110.5
3rdQu
27999.2
Max
55961.9
,
7exmo
Min
133.399
1stQu
153.668
Median
165.727
Mean
175.552
3rdQu
188.343
Max
266.591
,
8gemini
Min
998.186
1stQu
2098.88
Median
2581.5
3rdQu
3024.17
Mean
3048.28
Max
7406.78
,
9kraken
Min
3187.55
1stQu
5765.37
Median
6522.84
Mean
7749.91
3rdQu
9236.13
Max
17022.8
,
10others
30
,
11wallofcoins
23
0.00177723
1
0.0146006
1
0.0253634
1
0.029009
1
0.0360106
1
(Other)
2
Plots
Price data
Here we extract the non-time columns in the tabular price data obtained above and plot the corresponding time series:
DateListPlot[Association[#->Normal[dsBTCPriceData[All,{"Time",#}][Values]]&/@Rest[Normal[Keys[dsBTCPriceData〚1〛]]]],AspectRatio1/4,ImageSizeLarge,PlotLabel->Row[{"BTC-EUR,coinbase"}],PlotTheme"Detailed"]
Out[]=
May24
May31
Jun07
Jun14
26000
28000
30000
32000
34000
36000
BTC-EUR,coinbase
avg
max
min
Volume data
Here we extract the non-time columns (corresponding to exchanges) in the tabular volume data obtained above and plot the corresponding time series:
In[]:=
DateListPlot[Association[#->Normal[dsBTCVolumeData[All,{"Time",#}][Values]]&/@Rest[Normal[Keys[dsBTCVolumeData〚1〛]]]],PlotRange->All,AspectRatio1/4,ImageSizeLarge,PlotLabel->Row[{"BTCvolume"}],PlotTheme"Detailed",GridLinesAutomatic]
Out[]=
May24
May31
Jun07
Jun14
0
10000
20000
30000
40000
50000
60000
70000
BTCvolume
bit-x
bitfinex
bitstamp
cex.io
coinbase
exmo
gemini
kraken
others
wallofcoins
|
