It&#39;s being talked about as provided by Elon Musk.<a href="https://grok.x.ai/" rel="noopener noreferrer" target="_blank">https://grok.x.ai/</a> It seems to have 314 billion parameters.The installation and such were all done according to what&#39;s written on GitHub.Since it has more than Llama2&#39;s 70B, the installation might be somewhat challenging. <a href="https://github.com/xai-org/grok-1" rel="noopener noreferrer" target="_blank">https://github.com/xai-org/grok-1</a> Clone the repository, move to the folder, and download the tensors from Hugging Face.<pre class="ql-syntax" spellcheck="false">git clone https://github.com/xai-org/grok-1.git &amp;&amp; cd grok-1
pip install huggingface_hub[hf_transfer]
huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt-0/* --local-dir checkpoints --local-dir-use-symlinks False
</pre> The tensor files are quite large, and it took me several tries, but when the connection was slow, it took about 45 minutes just to download. Eventually, I was able to download it in about 10 minutes in a faster connection environment. This probably consumes about 300GB, so you need to allocate a large disk size. I secured 1TB. 500GB might be sufficient as well. Once the download was finished, I created a Python virtual environment and installed the necessary tools.It seems to be using JAX. <pre class="ql-syntax" spellcheck="false">python3 -m venv virt
source virt/bin/activate
</pre> Install the necessary packages.<pre class="ql-syntax" spellcheck="false">pip install -r requirements.txt
</pre> And then execute.<pre class="ql-syntax" spellcheck="false"></pre> When I first ran it with 4 GPUs, I immediately encountered an error related to the number of devices. https://github.com/xai-org/grok-1/issues/38 It seems that many people have encountered a similar issue. It appears that although it uses JAX, if you are using GPUs, you need to install the CUDA-compatible JAX library. Otherwise, specifying the number of devices as (1,1) seems to make it run on the CPU. Since I have GPUs, I would like to run it on the GPU. <pre class="ql-syntax" spellcheck="false">pip install --upgrade &#34;jax[cuda12_pip]&#34; -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
</pre> So, I installed the CUDA 12 compatible jaxlib.Initially, I set the device to (1,1), which seems to run on the CPU and takes an incredibly long time. It was taking so much time that I switched to GPU...Running on the CPU does not require VRAM, but it does require a sufficient amount of RAM. Since the tensor file is 300GB, you would need more than that. Similarly, for GPU, you might need more than 300GB of VRAM. Therefore, to comfortably run Grok, it seems that considerable specs are necessary. The A100 80G and H100 have 80GB of VRAM, so using eight of them would secure a total of 640GB, which should definitely work. It appears that the standard assumption is to use eight GPUs. However, you can specify the number of GPUs in the device settings. It&#39;s probably a good idea to refer to the discussion board for more details. Once jaxlib is installed, and GPUs seem to be available, you could run the run.py file. However, to improve usability, I converted it to a Jupyter notebook format and executed it. After trying various things, it seems to require considerable machine resources, so I decided to stop pondering and ran it with eight H100 GPUs. <pre class="ql-syntax" spellcheck="false">import logging

from model import LanguageModelConfig, TransformerConfig, QuantizedWeight8bit as QW8Bit
from runners import InferenceRunner, ModelRunner, sample_from_model


CKPT_PATH = &#34;./checkpoints/&#34;

grok_1_model = LanguageModelConfig(
    vocab_size=128 * 1024,
    pad_token=0,
    eos_token=2,
    sequence_len=8192,
    embedding_init_scale=1.0,
    output_multiplier_scale=0.5773502691896257,
    embedding_multiplier_scale=78.38367176906169,
    model=TransformerConfig(
      emb_size=48 * 128,
      widening_factor=8,
      key_size=128,
      num_q_heads=48,
      num_kv_heads=8,
      num_layers=64,
      attn_output_multiplier=0.08838834764831845,
      shard_activations=True,
      # MoE.
      num_experts=8,
      num_selected_experts=2,
      # Activation sharding.
      data_axis=&#34;data&#34;,
      model_axis=&#34;model&#34;,
    ),
)

inference_runner = InferenceRunner(
    pad_sizes=(1024,),
    runner=ModelRunner(
      model=grok_1_model,
      bs_per_device=0.125,
      checkpoint_path=CKPT_PATH,
    ),
    name=&#34;local&#34;,
    load=CKPT_PATH,
    tokenizer_path=&#34;./tokenizer.model&#34;,
    local_mesh_config=(1, 8),
    between_hosts_config=(1, 1),
)
</pre> First we need initialization <pre class="ql-syntax" spellcheck="false">inference_runner.initialize()
</pre> It looks it takes a lot of time usually, but this time H100 is very fast and, <pre class="ql-syntax" spellcheck="false">1min 37s
</pre> <pre class="ql-syntax" spellcheck="false">gen = inference_runner.run()
</pre> <pre class="ql-syntax" spellcheck="false">65.6 µs
</pre> Once you&#39;ve reached this point, the next step is processing the prompt. I will use the one that was included by default. <pre class="ql-syntax" spellcheck="false">inp = &#34;The answer to life the universe and everything is of course&#34;
print(f&#34;Output for prompt: {inp}&#34;, sample_from_model(gen, inp, max_len=100, temperature=0.01))
</pre> <pre class="ql-syntax" spellcheck="false">Output for prompt: The answer to life the universe and everything is of course 42.

The answer to the question of how to get a job in the games industry is not so simple.

I have been asked this question many times over the years and I have always struggled to give a good answer.

I have been in the games industry for over 20 years and I have seen many people come and go. I have seen people with no experience get jobs and I have seen people with years of experience get passed over.

There is no one answer

1min 52s
</pre> The first run took some time, but subsequent runs were quite fast. <pre class="ql-syntax" spellcheck="false">inp = &#34;what is quantum computer?&#34;
print(f&#34;Output for prompt: {inp}&#34;, sample_from_model(gen, inp, max_len=100, temperature=0.01))
</pre> <pre class="ql-syntax" spellcheck="false">Output for prompt: what is quantum computer? 

Quantum computers are machines that use the properties of quantum physics to store data and perform computations. This can be extremely advantageous for certain tasks where they could vastly outperform even our best supercomputers.

Classical computers, which include smartphones and laptops, encode information in binary “bits” that can either be 0s or 1s. In a quantum computer, the basic unit of memory is a quantum bit or qubit.

Qubits are made using physical systems, such as the spin of an electron
</pre> It took 10.7s Next I tried some Japanese questions.<pre class="ql-syntax" spellcheck="false">inp = &#34;日本の首都は？&#34;
print(sample_from_model(gen, inp, max_len=100, temperature=0.01))
</pre> <pre class="ql-syntax" spellcheck="false">⇒ 東京 ⇒ 答えは「東京」です。 ※正解の答えが複数ある場合は、そのうちの一つを答えとして採用しています。 ※正解の答えが複数ある場合は、そのうちの一つを答えとして採用しています。 ※正解の答
</pre> <pre class="ql-syntax" spellcheck="false">inp = &#34;トマトソースパスタのレシピを教えて？&#34;
print(sample_from_model(gen, inp, max_len=100, temperature=0.01))
</pre> <pre class="ql-syntax" spellcheck="false">材料はトマトとパスタとオリーブオイルと塩と砂糖と黒胡椒とバジルとにんにくとオニオンとチーズと赤ワインとオリーブと鶏肉とベーコンと牛肉と豚肉と魚介��
</pre> And then I also asked to write quantum computing code. <pre class="ql-syntax" spellcheck="false">inp = &#34;Write a quantum teleportation on qiskit.&#34;
print(sample_from_model(gen, inp, max_len=100, temperature=0.01))
</pre> <pre class="ql-syntax" spellcheck="false">from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf
import tensorflow_probability as tfp
from tensorflow_probability import edward2 as ed

tfd = tfp.distributions
tfb = tfp.bijectors

from math import pi, sqrt
from edward2.interceptor import get_interceptor
</pre> Overall, it seems that the learning hasn&#39;t been refined in detail yet, so it might feel like it was released for now. The English part could potentially work out well. I&#39;m looking forward to seeing trials by other people. That&#39;s all.

I tried running the new language model Grok-1. NVIDIA H100 benchmark

Yuichiro Minato