BEIS open data API manual

About the API

This API provides access to the data held by BEIS and previously only published in spreadsheet or pdf format. Over time, many data publications will appear here. The first data published is a subset of DUKES and an experimental lookup table of calorific values of various fuel types for different years. Over time, we will add more DUKES data, as well as other data publications.

REST stands for “Representational state transfer” which is an architectural style based on the standard HTTP commands, such as GET or POST, and means that the call to the API looks like a standard (if sometimes long) web address.

DUKES

The DUKES API presents the data from the official DUKES publication spreadsheet in a machine-readable format. It is accessible via a REST endpoint through a URL composed of the desired energy types, uses of the selected energy and parameters that allow selection by, for example, date range - see below for a full list of the selectable features.

Anatomy of an API request

The URL is made up of various elements:

http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=csv&year_start=2012&year_end=2014&orientation=narrow
hostroutetypequery

The host part is the address of the server on the internet.

The route represents the original data source. In this case, it is part of the energy data, and is the DUKES data set.

The type section contains your selected energy type of flow of energy. There should be at least one of each - if you don't want to limit your selection, set this to “energy/flows”. The complete list of energy types and flows is presented below - use the term given in the “id” column in this section. You can have as many of these as you need, in any order.

The query section allows you to filter your selection of energy types and flows, for instance by year. Parameters that control the output are also here, for instance file format or how the data is arranged. The complete list of parameters is also below.

Open each category by clicking the heading or icon.

Typical usage is illustrated given in the example panels

Type parameters

nameidno.
Flowsflows0
Supplysupply1
Indigenous productionproduction9
Primary electricityprimary_electricity15
Major power producerspe_major_power_producers16
Nuclearpe_major_power_producers_nuclear17
Large scale hydrope_major_power_producers_large_scale_hydro18
Small scale hydrope_major_power_producers_small_scale_hydro19
Wind and solarpe_major_power_producers_wind_and_solar20
Other generatorspe_other_generators21
Nuclearpe_other_generators_nuclear22
Large scale hydrope_other_generators_large_scale_hydro23
Small scale hydrope_other_generators_small_scale_hydro24
Wind, wave and solar photovoltaicspe_other_generators_wind_wave_and_solar_photovoltaics25
Secondary electricitysecondary_electricity26
Major power producersse_major_power_producers27
Coalse_major_power_producers_coal28
Oilse_major_power_producers_oil29
Gasse_major_power_producers_gas30
Renewablesse_major_power_producers_renewables31
Otherse_major_power_producers_other32
Other generatorsse_other_generators33
Coalse_other_generators_coal34
Oilse_other_generators_oil35
Gasse_other_generators_gas36
Renewablesse_other_generators_renewables37
Otherse_other_generators_other38
Primary and secondary productionprimary_and_secondary_production39
Nuclearprimary_and_secondary_production_nuclear40
Hydroprimary_and_secondary_production_hydro41
Wind, wave and solar photovoltaicsprimary_and_secondary_production_wind_wave_and_solar_photovoltaics42
Coalprimary_and_secondary_production_coal43
Oilprimary_and_secondary_production_oil44
Gasprimary_and_secondary_production_gas45
Other renewablesprimary_and_secondary_production_other_renewables46
Otherprimary_and_secondary_production_other47
Other sourcesother_sources49
Importsimports50
Exportsexports51
Marine bunkersmarine_bunkers52
Stock changestock_change53
Transferstransfers_supply54
Total supplytotal_supply55
Statistical differencestatistical_difference57
Total demandtotal_demand58
Transferstransfers_demand80
Transformationtransformation110
Electricity generationtransformation_electricity_generation111
Major power producersmajor_power_producers113
Autogeneratorsautogenerators114
Heat generationheat_generation115
Petroleum refineriestransformation_petroleum_refineries116
Coke manufacturetransformation_coke_manufacture117
Blast furnacestransformation_blast_furnaces118
Patent fuel manufacturetransformation_patent_fuel_manufacture119
Patent fuel manufacture and low temperature carbonisationtransformation_patent_fuel_manufacture_and_low_temperature_carbonisation120
Othertransformation_other123
Energy industry useenergy_industry_use124
Electricity generationenergy_industry_use_electricity_generation126
Oil and gas extractionoil_and_gas_extraction127
Petroleum refineriesenergy_industry_use_petroleum_refineries128
Coal extractionenergy_industry_use_coal_extraction129
Coke manufactureenergy_industry_use_coke_manufacture130
Coal extraction and coke manufactureenergy_industry_use_coal_extraction_and_coke_manufacture131
Blast furnacesenergy_industry_use_blast_furnaces132
Patent fuel manufactureenergy_industry_use_patent_fuel_manufacture133
Pumped storagepumped_storage134
Otherenergy_industry_use_other135
Losseslosses137
Final consumptionfinal_consumption141
Industryindustry143
Unclassifiedunclassified~industry145
Iron and steeliron_and_steel146
Non-ferrous metalsnon-ferrous_metals147
Mineral productsmineral_products148
Chemicalschemicals149
Mechanical engineering etcmechanical_engineering150
Electrical engineering etcelectrical_engineering151
Vehiclesvehicles152
Food, beverages etcfood_beverages153
Textiles, leather etctextiles_leather154
Paper, printing etcpaper_printing155
Other industriesother_industries156
Constructionconstruction157
Transporttransport~final_consumption158
Airair176
Railrail177
Roadroad178
National navigationnational_navigation179
Pipelinespipelines180
Otherfinal_consumption_other181
Domesticdomestic183
Public administrationpublic_administration190
Commercialcommercial193
Agricultureagriculture201
Miscellaneousmiscellaneous203
Non energy usenon_energy_use224
nameidno.
Energyenergy0
Coalcoal1
Manufactured fuelmanufactured_fuel2
Primary oilsprimary_oils3
Petroleum productspetroleum_products4
Natural gasnatural_gas5
Bioenergy and wastebioenergy_and_waste6
Primary electricityprimary_electricity7
Electricityelectricity_aggregate8
Heat soldheat_sold9
Aggregate energy balanceaggregate_energy_balance10
Steam coalsteam_coal11
Coking coalcoking_coal12
Anthraciteanthracite13
Total coaltotal_coal14
Crude oilcrude_oil15
Ethaneethane16
Propanepropane17
Butanebutane18
Condensatecondensate19
NGLngl20
Feedstockfeedstock21
Total primary oilprimary_oil22
Ethaneethane~petroleum_products23
Propanepropane~petroleum_products24
Butanebutane~petroleum_products25
Other gasesother_gases26
Naphthanaphtha27
Aviation spiritaviation_spirit28
Motor spiritmotor_spirit29
White spirit and SBPwhite_spirit_and_sbp30
Aviation turbine fuelaviation_turbine_fuel31
Burning oilburning_oil32
DERVderv33
Gas oilgas_oil34
Gas and Diesel oilgas_and_diesel_oil35
Fuel oilsfuel_oils36
Lubricantslubricants37
Bitumenbitumen38
Petroleum cokepetroleum_coke39
Miscellaneous Productsmisc_products40
Total petroleum productstotal_petroleum_products41
Natural gasnatural_gas~natural_gas42
Colliery methanecolliery_methane43
Total natural gastotal_natural_gas44
Electricityelectricity45
Waste woodwaste_wood46
Woodwood47
Animal biomass and anaerobic digestionanimal_biomass_anaerobic_digestion48
Plant biomassplant_biomass50
Sewage gassewage_gas51
Landfill gaslandfill_gas52
Wastewaste53
Solar photovoltaics - active solar heating - and deep geothermalsolar_photovoltaics_active_solar_heating_deep_geothermal54
Heat pumpsheat_pumps56
Hydrohydro57
Wind and marine energywind_and_marine_energy59
Liquid biofuelsliquid_biofuels60
Total renewablestotal_renewables61

Query parameters

The list of query parameters is appended to the very end of the URL after a single '?'. Each query parameter is separated from other query parameters with a single '&' character.

Allowable values are listed between '[' and ']', separated by '|'. The default value is given in bold.

  • type string
  • values [json | csv]
  • operation affects output
  • multiple items? no
  • type string
  • values [GWh | ktoe | mtonnes | original | all]
  • operation affects items selected
  • multiple items? no
  • type string
  • values [narrow | wide]
  • operation affects output data format/visual appearance
  • multiple items? no
  • type integer
  • minimum 1998
  • maximum most recent year data is available
  • default most recent year
  • operation affects items selected
  • multiple items? no
  • type string
  • values [balance | trade | cv | supply | deliveries | distribution | fuelused | renewable | powerstations | capacity | generation]
  • operation affects items selected
  • multiple items? no
  • type integer
  • minimum 1
  • default ALL
  • operation affects items selected
  • multiple items? no
  • type string
  • values [normal | all | suppressed | revised]
  • operation affects items selected
  • multiple items? no

Calorific values (experimental)

This lookup table is experimental and is accessible through its own REST endpoint, as shown below. The URL is composed of the desired energy type and desired year. This returns a single value expressed in GJ/tonne.

Anatomy of an API request

The URL is made up of various elements:

http://njs.analysisoncbas.co.uk/fuel/cv/ethane/2015
hostroutetype

The host part is the address of the server on the internet.

The route represents the original data source. In this case, it is part of the fuel data, and is the cv data set. An alternative version is /fuel/calorific_value

The type section contains your selected fuel type and they desired year. There should be one of each in this order.

Fuel types

nameidno.
Ethaneethane16
Propanepropane17
Butanebutane18
Other gasesother_gases26
Naphthanaphtha27
Aviation spiritaviation_spirit28
Motor spiritmotor_spirit29
White spirit and SBPwhite_spirit_and_sbp30
Aviation turbine fuelaviation_turbine_fuel31
Burning oilburning_oil32
DERVderv33
Gas oilgas_oil34
Fuel oilsfuel_oils36
Lubricantslubricants37
Bitumenbitumen38
Petroleum cokepetroleum_coke39
Total petroleum productstotal_petroleum_products41
Coke oven cokecoke_oven_coke82
Coke breezecoke_breeze83
Other manufactured solid fuelsother_manufactured_solid_fuels84
Petroleum waxpetroleum_wax107
Industrial coalindustrial_coal108

FAQ

  • My primary need is for modeling. I am trying to get the data into my model: how can using the API be made to be more efficient and simple to incorporate into my model?

    We have considered the needs of modelers, and are working to introduce features that are specific to your needs. For instance, we will be keeping the entire dataset for previous years, so you can re-run your (old) model and produce the same results, though obviously this will bypass any figures we may have had to revise in the interim.

  • Can I have pre-prepared tables, and if so what how/where/what...?

    We have no plans to produce any pre-prepared tables in the same way that some secondary DUKES spreadsheets take primary data and represent it in a different format. We believe that this defeats the purpose of providing access to the data in a much more flexible form. Once you have worked out exactly how you want your data, you only need to keep the URL with the appropriate parameters in order to re-use it. What we may introduce is a set of pre-calculated aggregate data, but we need to learn more about what would be of benefit, and to whom.

  • How can I identify the cell X that I am used to using...?

    Individual cells are locatable, but you may have to think about how to uniquely identify it in terms of the energy type, flow type, year and other relevant characteristics. Although the source table is provided, the intention is that this is more for reassurance for those familiar with the existing spreadsheet structure whilst this system is being developed and should not be relied upon as a means of location. It may be phased out at some point in the future.

  • How can I be certain that the API is giving me exactly what I am used to getting from DUKES table x.x? Can I trust the data?

    Yes. We have quality assured the data during the transfer process, and all the values have been validated as being correct.

  • Why are there so many different units used? It makes comparisons difficult. Can't data be presented in the units I want?

    The variety of units used is because different energy industry sectors traditionally use units appropriate to their industry - for instance it makes sense to measure coal in tonnes, but not electricity. It is possible for the API to convert units, but this functionality was not built into this release of the API because it isn't always a straightforward calculation due to variations in calorific value, even in the same fuel across different years. It is something we are considering doing, but if we decide to implement it, will require a lot of testing and quality control, so won't be launched in the foreseeable future.

Code examples

Here are some examples of how you might programmatically access the data.

This URL used in the code examples uses DUKES. It selects the total supply of electricity from 2012 to 2014 inclusive:
http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=csv&unit=original&year_start=2012&year_end=2014&detail=full&orientation=narrow

Javascript


// this assumes jQuery and/or d3 have been loaded and are available through $ or d3 variables

var URL_str = "http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=csv&unit=original&year_start=2012&year_end=2014&detail=full&orientation=narrow";

// 1.1 using jQuery Ajax, specifying csv in the header
$.ajax({url: URL_str, accepts: {csv: 'text/csv'}, dataType: 'text'}).done(function( data ) {
	console.log(data);
});

// 1.2 using jQuery convenience function for JSON (there isn't a corresponding CSV function)
$.getJSON( URL_str, function( data ) {
	console.log(data);
});


// 2. using d3 convenience function for CSV (there are also JSON and generic Ajax functions)
d3.csv(URL_str, function(error, data) {
	if (error) throw error;
	console.log(data);
}
				

Python (3 & 2.7)


# this uses the requests package - there are other packages available for accessing http data
import requests

URL_str = "http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=csv&unit=original&year_start=2012&year_end=2014&detail=full&orientation=narrow"

# using requests
r = requests.get(URL_str, headers={"accept":"text/csv"})
print(r.text)  # basic output to console

# item,display_order,fuel,year,energy,unit,src
# Primary supply,55,045:Electricity,2012,375737.48,GWh,5.1
# Primary supply,55,045:Electricity,2013,372714.23,GWh,5.1
# Primary supply,55,045:Electricity,2014,358615.89,GWh,5.1
				

R


# this uses the httr library - there are other libraries available for accessing http data

library(httr)
library(readr)

URL_str <- "http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=csv&unit=original&year_start=2012&year_end=2014&detail=full&orientation=narrow"

csv <- GET(URL_str, accept("text/csv"))
content(csv) # uses readr to display data neatly with added metadata

# A tibble: 3 x 7
#             item display_order            fuel  year   energy  unit   src
#                                       
# 1 Primary supply            55 045:Electricity  2012 375737.5   GWh   5.1
# 2 Primary supply            55 045:Electricity  2013 372714.2   GWh   5.1
# 3 Primary supply            55 045:Electricity  2014 358615.9   GWh   5.1

				

Bash (Terminal commands in Unix/Linux/MacOS)


# remember to include entire URL in quotes, otherwise the '&' is interpreted as a line break

# this outputs to stdout (i.e. this console)
curl "http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=csv&unit=original&year_start=2012&year_end=2014&detail=full&orientation=narrow"
# item,display_order,fuel,year,energy,unit,src
# Primary supply,55,045:Electricity,2012,375737.48,GWh,5.1
# Primary supply,55,045:Electricity,2013,372714.23,GWh,5.1
# Primary supply,55,045:Electricity,2014,358615.89,GWh,5.1

# this outputs to a file called electricity_totalsupply.csv
curl "http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=csv&unit=original&year_start=2012&year_end=2014&detail=full&orientation=narrow" -o electricity_totalsupply.csv

# this outputs in wide orientation - the same query as above, but pivoted. Note how not all columns are output
curl "http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=csv&unit=original&year_start=2012&year_end=2014&detail=full&orientation=wide"
# item,year,045:Electricity
# Primary supply,2012,375737.48
# Primary supply,2013,372714.23
# Primary supply,2014,358615.89

# and this outputs in json format
curl "http://njs.analysisoncbas.co.uk/energy/DUKES/electricity/total_supply?format=json&unit=original&year_start=2012&year_end=2014&detail=full&orientation=wide"
# [{"item":"Primary supply","year":2012,"045:Electricity":375737.48},{"item":"Primary supply","year":2013,"045:Electricity":372714.23},{"item":"Primary supply","year":2014,"045:Electricity":358615.89}]