Llama2の2つのシステムについて、ちょっと以前は違いがわからなかったんですが、聞きたかった内容をChatの方で実行してみたら意外と良かったので70B（700億パラメータ）のモデルを量子化などの軽量化なしでVRAM256GBで素で実行してみたいと思います。手元に使える8GPU-NVLINKのマシンがあってそれを使ってみました。 モデルをダウンロードしたら、 <pre class="ql-syntax" spellcheck="false">torchrun --nproc_per_node 8 example_chat2.py   --ckpt_dir llama-2-70b-chat/   --tokenizer_path tokenizer.model   --max_seq_len 512 --max_batch_size 6
</pre> で実行できます。MP=8を指定し、ファイルはちょっと日本語を入れたかったのでファイル名を改造して中のプロンプトをちょっと変えました。モデルは70B(700億)パラメータを指定します。 日本語で聞いたのですが、英語できましたが、結果は7Bや13Bと比べるとやはり満足できる物でした。 <pre class="ql-syntax" spellcheck="false">Loaded in 87.09 seconds
User: カレーライスのレシピは？

&gt; Assistant: Sure! Here&#39;s a simple recipe for curry rice:

Ingredients:

* 2 cups cooked rice
* 1 tablespoon vegetable oil
* 1 onion, diced
* 2 cloves garlic, minced
* 1 tablespoon grated ginger
* 1 teaspoon curry powder
* 1 teaspoon turmeric powder
* 1/2 teaspoon cumin powder
* 1/2 teaspoon coriander powder
* 1/4 teaspoon cayenne pepper (optional)
* 1 can diced tomatoes (14 oz)
* 1 cup water or vegetable broth
* Salt and pepper to taste
* Fresh cilantro leaves for garnish (optional)

Instructions:

1. Heat the oil in a large saucepan over medium heat. Add the onion, garlic, and ginger, and sauté until the onion is translucent.
2. Add the curry powder, turmeric powder, cumin powder, coriander powder, and cayenne pepper (if using), and stir for 1-2 minutes.
3. Add the diced tomatoes, water or vegetable broth, salt, and pepper. Stir well.
4. Bring the mixture to a boil, then reduce the heat to low and simmer for 10-15 minutes, stirring occasionally.
5. Fluff the cooked rice with a fork and add it to the saucepan. Stir well to combine with the curry sauce.
6. Cook for an additional 2-3 minutes, stirring frequently, until the rice is heated through and the curry sauce has thickened slightly.
7. Taste and adjust the seasoning as needed.
8. Garnish with fresh cilantro leaves (if using) and serve hot.

Enjoy your delicious curry rice!
</pre> 7Bとかだとキャロットライスとか聞き間違いがありましたので。その他は元の例題ファイルそのものを実行してみました。 <pre class="ql-syntax" spellcheck="false">User: I am going to Paris, what should I see?

Assistant: Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:

1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.
2. The Louvre Museum: The Louvre is one of the world&#39;s largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.
3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.

These are just a few of the many attractions that Paris has to offer. With so much to see and do, it&#39;s no wonder that Paris is one of the most popular tourist destinations in the world.

User: What is so great about #1?

&gt; Assistant: The Eiffel Tower is a legendary symbol of Paris and one of the most recognizable landmarks in the world. Here are some reasons why it&#39;s a must-see attraction:

1. Iconic status: The Eiffel Tower is an iconic symbol of Paris and France, and it&#39;s one of the most photographed landmarks in the world.
2. Engineering marvel: The Eiffel Tower was built in 1889 for the World&#39;s Fair, and it was a groundbreaking feat of engineering at the time. It stands 324 meters (1,063 feet) tall and was the tallest structure in the world for 41 years.
3. Panoramic views: The Eiffel Tower offers breathtaking views of Paris from its observation decks. You can see the city&#39;s famous landmarks, such as the Arc de Triomphe, Notre-Dame Cathedral, and the Seine River.
4. Romantic atmosphere: The Eiffel Tower has a romantic atmosphere, especially at night when it&#39;s lit up with thousands of twinkling lights. It&#39;s a popular spot for couples and proposal spot.
5.

==================================

System: Always answer with Haiku

User: I am going to Paris, what should I see?

&gt; Assistant: Eiffel Tower high
Art museums, fashion, and wine
Paris, city of love

==================================

System: Always answer with emojis

User: How to go from Beijing to NY?

&gt; Assistant: 🚗🗺️🛬🏙️

==================================

System: You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don&#39;t know the answer to a question, please don&#39;t share false information.

User: Write a brief birthday message to John

&gt; Assistant: &#34;Dear John, I hope your birthday is filled with joy, love, and all your favorite things. May this year bring you happiness, success, and countless moments to cherish. Wishing you a wonderful day and a bright future! 🎉🎂❤️&#34;
</pre> なんかいい気がします。手元の8GPUシステムは他にも1台あるので、一台はLLM専用で使ってみたいと思います。以上です。

Llama2の70B-Chatを8GPUシステムで実行

Yuichiro Minato