Folclore Brasileiro Data

Working with folclore brasileiro data in practice

The first thing you need to understand is that there is no single authoritative database. Every dataset you find online is someone's interpretation of oral traditions that were never written down in any systematic way. I spent about fourteen months building a cleaned collection from disparate sources, and the hardest part wasn't technical at all. It was dealing with the fact that the same entity might be called Curupira, Carepi, or even Carepo depending on which region the source came from. The main repositories are scattered across government portals, university digitization projects, and a few private collections that are poorly documented. The IBGE has some regional surveys buried in their geography division publications. The Funag (Fundação Alexandre de Gusmão) occasionally releases cultural heritage datasets. Then there are the academic ones - the Museu do Folclore Edison Carneiro in São Paulo has material, though their digitization efforts are incomplete and uneven. Most researchers end up scraping from portals like reponet, scielo, and the digital archives of state universities with strong anthropology departments. A practical source I ended up relying on heavily was the catalog from the Programa de Memória Oral da Universidade Federal de Pernambuco. Their field recordings have metadata that is actually usable, unlike most university digital collections which are just PDFs without structure.

The data model problem nobody talks about

Here is where people get stuck. Brazilian folklore entities don't fit into relational schemas well. Take the Boto Cor-de-Rosa, for example. In one dataset it appears as a singular legend from the Northern region. In another, it overlaps with the Caboclo das Águas, which is sometimes treated as the same entity and sometimes as a distinct being. A third source calls it Malasqueira in parts of Pará. If you build a simple key-value table, you will generate duplicates or merge unrelated entries. I ended up using a property graph model. Each entity becomes a node with properties for canonical name, variant names, region, type (lenda, mito, festa, brinquedo, etc.), and source citations. Relationships connect variants to the canonical node with a confidence score based on how many independent sources corroborate the link. This added about three days of setup work but saved me weeks of cleaning later. The other insight that isn't obvious: temporal data matters more than people expect. Many folklore entities shifted meaning or geography over the twentieth century. A Curupira entry from a 1950s survey might describe the entity differently than one from the 1980s because the cultural narrative itself changed. If your dataset is time-agnostic, you lose that dimension entirely. I added a collection_date property to every record and flagged entries that showed significant semantic drift between periods.

What the data actually looks like when cleaned

A well-structured folclore brasileiro data entry typically contains about twelve to fifteen fields. The core ones are entity_name, entity_type, geographic_origin, description_summary, primary_sources, variant_names, related_entities, cultural_context, and last_verified_date. Beyond that you might have audio or visual references, transcription quality scores, and community attribution notes if the tradition comes from an identifiable local group. The description field is the most problematic. Most sources write descriptions at different levels of detail. Some are one paragraph. Others are book chapters scanned as PDF. I found that normalizing to a maximum of three sentences per description kept the data searchable without losing essential information. Anything longer gets truncated with a link to the full source.

Downloadable folclore brasileiro data resources

There is no centralized download link for a complete dataset. What exists are partial collections. The best starting point I found is a GitHub repository maintained by a researcher at UnB that aggregates metadata from about forty different sources. It is not complete by any standard, but the schema is reasonable and the license information is mostly clear. Another option is the open dataset from the Secretaria de Cultura do Estado da Bahia, which focuses on Afrobrazilian ceremonial traditions and has cleaner documentation than most government releases. If you need something ready to use immediately, the Kaggle collection titled Brazilian Folklore Legends contains roughly two thousand entries scraped from public websites. The quality is uneven. Approximately thirty percent of the entries have incomplete geographic data. You can work with it if you account for that gap.

Common failures and how to avoid them

The biggest mistake I see is treating this data as static. It isn't. New field recordings appear regularly. Regional classifications get revised. An entity that was categorized as a lenda in one survey might be reclassified as a mito in a later one by the same institution. If you are building something that depends on this data being current, you need a refresh pipeline, not a one-time download. Another issue is the language layer. Many older sources use archaic Portuguese spelling and regional vocabulary that doesn't map cleanly to modern NLP tools. Tokenizers trained on contemporary text will misparse names like Saci-Pererê or Iara in unexpected ways. I ran into this when trying to run named entity recognition across the Funag corpus. The model kept splitting compound names incorrectly. The workaround was building a small custom tokenizer with a domain-specific vocabulary list rather than fighting with a general-purpose model. A third failure mode is assuming numeric values are available when they aren't. Most folklore data is qualitative by nature. You will not find population statistics attached to entities or measurable cultural impact indices. Any attempt to force quantification onto this material usually produces misleading results. If your application requires numerical inputs, you need to create your own proxy metrics and be explicit about what they represent.

Building a basic ingestion pipeline

Here is what a working pipeline looks like. Start by collecting raw records from your source APIs or web scraping. Run a deduplication pass using fuzzy matching on entity names with a threshold around 0.85 Jaccard similarity. This catches obvious duplicates before they pollute the database. After deduplication, apply your schema normalization. Map incoming fields to your canonical structure, filling gaps with null values rather than making assumptions. The null entries are easier to fix later than incorrect guesses. Then run a validation pass that checks for internal consistency. Geographic origins should map to recognized Brazilian regions. Entity types should come from your allowed taxonomy. Source URLs should be reachable or marked as archived. This step caught about eight percent of records that had corrupt metadata from the original sources. Fixing it before insertion saved significant downstream work. The final step is adding confidence scores. Every record should have a provenance score based on how many independent sources confirm it and how recent those sources are. A Curupira entry from a 1992 academic paper with three corroborating sources scores higher than a 2018 blog post with no citations. This scoring system lets consumers of your data weigh records appropriately without needing to inspect every source manually.

The whole process, from raw collection to validated dataset, takes roughly two weeks for a moderate-sized collection of around five thousand entities. That is if you are doing it carefully. Rushing it produces noise that is very expensive to clean later. I learned that from doing it twice.