README

Dolly

Databricks’ Dolly is an instruction-following large language model trained on the Databricks machine learning platform
that is licensed for commercial use. Based on pythia-12b, Dolly is trained on ~15k instruction/response fine tuning records
databricks-dolly-15k generated
by Databricks employees in capability domains from the InstructGPT paper, including brainstorming, classification, closed QA, generation,
information extraction, open QA and summarization. dolly-v2-12b is not a state-of-the-art model, but does exhibit surprisingly
high quality instruction following behavior not characteristic of the foundation model on which it is based.

Databricks is committed to ensuring that every organization and individual benefits from the transformative power of artificial intelligence. The Dolly model family represents our first steps along this journey, and we’re excited to share this technology with the world.

The model is available on Hugging Face as databricks/dolly-v2-12b.

 Model Overview

dolly-v2-12b is a 12 billion parameter causal language model created by Databricks that is derived from
EleutherAI’s Pythia-12b and fine-tuned
on a ~15K record instruction corpus generated by Databricks employees and released under a permissive license (CC-BY-SA)


 Known Limitations

 Performance Limitations
dolly-v2-12b is not a state-of-the-art generative language model and, though quantitative benchmarking is ongoing, is not designed to perform
competitively with more modern model architectures or models subject to larger pretraining corpuses.

The Dolly model family is under active development, and so any list of shortcomings is unlikely to be exhaustive, but we include known limitations and misfires here as a means to document and share our preliminary findings with the community.
In particular, dolly-v2-12b struggles with: syntactically complex prompts, programming problems, mathematical operations, factual errors,
dates and times, open-ended question answering, hallucination, enumerating lists of specific length, stylistic mimicry, having a sense of humor, etc.
Moreover, we find that dolly-v2-12b does not have some capabilities, such as well-formatted letter writing, present in the original model.

 Dataset Limitations
Like all language models, dolly-v2-12b reflects the content and limitations of its training corpuses.

- The Pile: GPT-J’s pre-training corpus contains content mostly collected from the public internet, and like most web-scale datasets,
it contains content many users would find objectionable. As such, the model is likely to reflect these shortcomings, potentially overtly
in the case it is explicitly asked to produce objectionable content, and sometimes subtly, as in the case of biased or harmful implicit
associations.

- databricks-dolly-15k: The training data on which dolly-v2-12b is instruction tuned represents natural language instructions generated
by Databricks employees during a period spanning March and April 2023 and inclu