Publishing

openbasket is an aggregator and a quick-lookup tool. It holds no data of its own: every fund, every row and every page says which public source it came from, with a link, an as-of date and the source’s credit line. It is not investment advice and is not affiliated with any fund issuer.

The scheduled build runs two pipelines after the usual ones:

kedro run --pipeline=universe
kedro run --pipeline=publish --params=publish.release_tag=2026-11

universe looks through every fund openbasket knows in one batch; publish turns the result into the static JSON API described in Data API and the release files. Neither runs as part of a plain kedro run.

The universe

Every fund in the issuer lists (issuer_products: iShares UK and US, SPDR US and UCITS) and every N-PORT series with a latest filing (fund_master).

One fund, many ISINs

Share classes of one issuer portfolio are one fund. The largest share class by net assets gives the fund its id; the others are aliases. An N-PORT series’ share-class ISINs are those other funds hold whose issuer LEI is the series’ LEI and whose issuer type is a registered fund (security_master). A series with no known ISIN is keyed by its series ID.

One fund, two sources

A US ETF that is both an issuer fund and an N-PORT series is one fund, keyed by the issuer’s ISIN. The series is matched by ISIN first, then, for US issuer sites only, by ticker (UCITS tickers are local codes). The series ID stays on the fund, so its N-PORT holdings remain a fallback, and series_match says how it was found.

Every ISIN names one fund; issuer funds claim theirs first. On 2026 Q3 N-PORT and the 3 November 2026 issuer lists the universe is 14,870 funds: 1,913 iShares, 362 SPDR and 12,595 N-PORT series, with 605 US issuer ETFs matched to their series (261 by ISIN, 344 by ticker).

Running it

Each fund is expanded with the same engine and providers as the lookthrough pipeline, named by its fund ref rather than matched from an ISIN.

• A fund that raises is recorded with its error and a short traceback, and the batch moves on. Warnings logged while a fund is expanded (an issuer file that could not be read, say) are kept in its warnings column, and a fund with no holdings says why in its note.

• Each fund’s time is recorded in seconds; progress is logged every universe.log_every funds with an estimate of the time left.

• Requests to one host are spaced at least universe.http.default_interval_s apart (1 s), whichever client makes them; per_host_interval_s overrides it per host. Issuer files are cached per fund per day, so a rerun on the same day makes no requests.

• Once a source resolves lookthrough.min_resolved_pct, later sources are not read (universe.stop_early). TickerInside is asked only for tickers its fund list puts in the issuer layer (universe.tickerinside_prefilter).

• Which N-PORT series issues an ISIN is looked up in an index built with one scan of N-PORT; a series’ own lines are read by series ID from nport_holdings, which is sorted by series ID.

• OpenFIGI names and types are added once, for every constituent ISIN of the run, through the shared cache, which is saved every universe.openfigi_save_every funds and at the end.

• Constituent rows are written to Parquet parts under universe.work_dir as they are made, so memory does not grow with the universe.

Quick runs

Parameter	Effect
universe.sources	Fund lists to include: ishares, spdr, nport. A fund belongs to the list it is keyed by.
universe.ids	Only these fund ids or series IDs.
universe.sample	This many funds per list, chosen with universe.seed.
universe.limit	Then only the first this many.

kedro run --pipeline=universe --params=universe.sample=14   # 42 funds

Outputs

universe (data/03_primary/universe.parquet)

The whole universe before selection: id, isin, isins, series_id, issuer_id, name, ticker, issuer, universe_source, series_match.

universe_funds (data/07_model_output/universe_funds.parquet)

One row per fund run: the columns above plus source, source_url, source_ref, credit, as_of, matched_via, note, sources_used, sources_tried, reported_total_pct, resolved_pct, unresolved_pct, unidentified_lines_pct, n_constituents, error, traceback, warnings and seconds. source is the source of the fund’s own holdings, or the list the fund came from when no source had any. credit joins the credit lines of every source its rows came from, its own first.

universe_constituents (data/07_model_output/universe_constituents.parquet)

Every constituent row of every fund, with fund_id, and one unresolved-weight row per fund. Columns as in Output, plus source_url and name_source (who supplied the name: the holdings source, or openfigi). Unlike lookthrough.csv, a row never mixes sources: an ISIN held directly according to N-PORT and through a sub-fund according to that sub-fund’s issuer file is two rows, each with its own source, as_of and via_fund.

universe_run (data/08_reporting/universe_run.json)

Start and end, seconds, funds attempted, succeeded, failed and without holdings, rows written, HTTP requests and time spent waiting.

Publishing

publish writes, under publish.site_dir (default build/site, not committed), api/v1/meta.json, api/v1/index.json, one api/v1/funds/{id}.json per fund and an {"alias_of": id} file for every other share-class ISIN; and under publish.release_dir (default build/release) the three release files. Funds whose run failed are left out of the API but kept, with their error, in openbasket-funds.parquet.

Sources

publish.sources in conf/base/parameters.yml holds each source’s name, link, licence and credit line. meta.sources lists every source that contributed a row or a fund, with the number of funds and the range of as-of dates; a source missing from the configuration is still listed, by its id, with a warning. Each fund file’s credit names every source its rows came from, and each constituent row carries its own source and as_of.

Release links

publish.release_tag (default: the current UTC year and month) and publish.repo (omaxim/openbasket) give release_url and full_data_url, the download link of openbasket-lookthrough.parquet.

Size

Fund files keep their publish.max_constituents (1,000) heaviest rows and say truncated: true when rows were cut. If api/v1 still comes out larger than publish.site_budget_mb (800), the cut is lowered, never below publish.min_constituents (50), and the large funds are rewritten. Every cut is logged and recorded in the quality report; no row is dropped from the release file.

quality-report.json

Funds listed, attempted, succeeded, failed, without holdings and published per list; funds by holdings source, by how they were matched and by how their series was found; the resolved-weight distribution; seconds per fund by list; the most common errors and warnings, with example funds; every fund whose source’s lines add up to more than 5 points away from 100%; API size and cuts; release file sizes; and the run record. It is also saved as data/08_reporting/quality_report.json.

Sample run

Run on 3 November 2026 with universe.sample=14 (42 funds, 14 per list), 2026 Q3 N-PORT and that day’s issuer files, starting with no issuer files cached:

	iShares	SPDR	N-PORT
Funds attempted	14	14	14
Succeeded	14	14	14
No holdings	2	0	0
Seconds per fund, mean	0.98	0.93	0.25
Seconds per fund, median	0.96	0.94	0.03

• No fund failed. The two iShares funds without holdings are index mutual funds whose page lists only the ten largest lines.

• Of the 40 funds with holdings, 29 resolve 99 to 101% of their weight to ISINs, 6 resolve 90 to 99%, 1 resolves 50 to 90% (ProShares Short FTSE China 50, whose N-PORT lines add up to 74.77%) and 4 more than 101%, whose reported lines add up to more than their net assets (two core bond ETFs at 121% and 120%). The median is 99.89%.

• The run took 57 s: 30 s expanding funds, of which 12 s waiting on the throttle, and 24 s mapping 4,609 new constituent ISINs through OpenFIGI. publish took under a second and wrote 1.9 MB of API (42 fund files, 5 cut to 1,000 rows), a 0.8 MB look-through file and a 27 kB fund file.

• A rerun the same day, with every issuer file cached, took 5 s.

A larger sample of 300 funds (universe.sample=100, universe.seed=1) gives steadier timings: 1.61 s per iShares fund (median 1.00, slowest 23 s for a large bond fund), 0.95 s per SPDR fund and 0.36 s per N-PORT series (median 0.03). 28 of the 100 iShares funds had no full holdings table. 271 funds had holdings; their median resolved weight was 99.82% and 252 resolved 90% or more. The API came to 12.8 MB, about 43 kB a fund. OpenFIGI mapped 31,220 new ISINs in 152 s, about 205 a second.

The full universe

From the 300-fund timings:

Step	Estimate
1,913 iShares funds	about 50 min (requests to one host are spaced 1 s apart)
362 SPDR funds	about 6 min
12,595 N-PORT series	30 to 75 min (most take 0.03 s; large funds and the 223 with a TickerInside ticker take longer)
OpenFIGI, first run	about 27 min for the 330,000 N-PORT ISINs not yet in the cache; minutes once the cache is kept between runs
publish	a few minutes

That is two to three hours for a first run and under two and a half once the OpenFIGI cache (data/03_primary/openfigi_cache.parquet) is kept between builds, well inside a six-hour job. The API should come to about 630 MB, under the 800 MB budget, so fund files would be cut only at 1,000 constituents.

Reference

Look through every fund openbasket knows, in one batch.

The universe is every fund in the issuer lists (iShares, SPDR) plus every N-PORT series with a latest filing. A US ETF that is both is one fund, keyed by the issuer’s ISIN, with the series kept so N-PORT can still serve as a fallback. Share classes of one portfolio are aliases of one fund.

Funds are expanded one at a time with the same engine and providers as the lookthrough pipeline. Unlike its output, a constituent row here is one ISIN from one source through one fund, so every row’s source is exact. A fund that raises is recorded with its error and the batch moves on. Constituent rows go to Parquet parts on disk as they are made, so memory stays flat however many funds there are.

openbasket.pipelines.universe.nodes.build_universe(issuer_products, fund_master, security_master, params)

Every known fund, one row each, with every ISIN that names it in isins.

A series joins an issuer fund when one of its share-class ISINs is the issuer fund’s ISIN, or, for US issuer sites, when one of its tickers is the issuer fund’s ticker. An ISIN belongs to one fund only: issuer funds claim theirs first.

Parameters:

• issuer_products (DataFrame)

• fund_master (DataFrame)

• security_master (DuckDBPyRelation)

• params (dict)

Return type:

DataFrame

openbasket.pipelines.universe.nodes.select_universe(universe, params)

The funds to run: sources, then ids, then sample funds per source, then limit.

Parameters:

• universe (DataFrame)

• params (dict)

Return type:

DataFrame

openbasket.pipelines.universe.nodes.fund_ref(fund)

The engine’s ref for a universe row; issuer funds keep their series for the N-PORT fallback.

Parameters:

fund (dict)

Return type:

FundRef

openbasket.pipelines.universe.nodes.expand_fund(fund, providers, params)

Look through one universe fund. Never raises: an error is returned in the summary.

Parameters:

• fund (dict)

• params (dict)

Return type:

tuple[dict, DataFrame]

openbasket.pipelines.universe.nodes.run_universe(universe, nport_holdings, security_master, fund_master, ticker_isin_map, issuer_products, cusip_isin_map, sedol_isin_map, lookthrough_params, params, sources)

Expand every selected fund. Returns the fund table, the constituent rows and a run record.

Parameters:

• universe (DataFrame)

• nport_holdings (DuckDBPyRelation)

• security_master (DuckDBPyRelation)

• fund_master (DataFrame)

• ticker_isin_map (DataFrame)

• issuer_products (DataFrame)

• cusip_isin_map (DuckDBPyRelation)

• sedol_isin_map (DataFrame)

• lookthrough_params (dict)

• params (dict)

• sources (dict)

Return type:

tuple[DataFrame, DuckDBPyRelation, dict]

Turn a universe run into the static JSON API and the release files.

The shapes are those in docs/source/data-api.md. Every fund file names its source, a link, the as-of date and a credit line; every constituent row names its own source and as-of date; meta.json lists every source that contributed a row, with its licence and credit, from publish.sources.

Fund files are cut to max_constituents rows, and if the API still comes out larger than site_budget_mb the cut is lowered and the large funds are rewritten. Each cut is logged and recorded in the quality report; the full lists are always in the release file.

openbasket.pipelines.publish.nodes.release_urls(repo, tag)

The release page and the full look-through file’s download link for tag.

Parameters:

• repo (str)

• tag (str)

Return type:

tuple[str, str]

openbasket.pipelines.publish.nodes.fund_document(fund, rows, n_rows, full_data_url)

One funds/{id}.json. rows are the kept constituent rows, n_rows how many there are in all.

Parameters:

• fund (dict)

• rows (list[tuple])

• n_rows (int)

• full_data_url (str)

Return type:

dict

openbasket.pipelines.publish.nodes.publish(universe, universe_funds, universe_constituents, universe_run, params)

Write api/v1 under site_dir and the release files under release_dir; return the quality report.

Parameters:

• universe (DataFrame)

• universe_funds (DataFrame)

• universe_constituents (DuckDBPyRelation)

• universe_run (dict)

• params (dict)

Return type:

dict

openbasket.pipelines.publish.nodes.quality_report(universe, funds, run, published)

Counts per source, resolution, timing, errors and funds whose lines do not add up to about 100%.

Parameters:

• universe (DataFrame)

• funds (DataFrame)

• run (dict)

• published (dict)

Return type:

dict

A process-wide politeness throttle for the sites openbasket reads.

Issuer sites are public web servers, not APIs with published limits. A universe run asks each of them for thousands of files, so every request to a host waits until at least min_interval seconds have passed since the last request to that host, whichever client made it.

openbasket.throttle.DEFAULT_INTERVAL_S = 0.5

Seconds between requests to one host unless configured otherwise.

openbasket.throttle.configure(default_s=None, per_host_s=None)

Set the intervals of the shared throttle and return it.

Parameters:

• default_s (float | None)

• per_host_s (dict[str, float] | None)

Return type:

Throttle